MLOps Demo

An end-to-end MLOps demo for educational use (EAISI). It combines a Dagster ML pipeline that trains an averaging ensemble on the Ames Housing dataset with a FastAPI inference service packaged as a Docker container for deployment to Scaleway Serverless Containers.

Architecture

AmesHousing.csv
      │
      ▼
┌─────────────────────────────────────────┐
│           Dagster Pipeline              │
│                                         │
│  raw_housing_data                       │
│        │                                │
│  preprocessed_data                      │
│        │                                │
│  training_test_split                    │
│        │                                │
│  averaging_model ──► ridge.onnx         │
│        │         ──► lasso.onnx         │
│        │         ──► xgboost.onnx       │
│        │         ──► preprocessing_     │
│        │              config.json       │
│  model_evaluation                       │
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│       FastAPI Inference Service         │
│                                         │
│  POST /predict  ──► ensemble average    │
│  GET  /health                           │
└─────────────────────────────────────────┘
      │
      ▼
Scaleway Serverless Containers (port 8080)

Models: Ridge + LassoCV + XGBoost simple average, exported to ONNX.
Target: log-transformed SalePrice; predictions are exponentiated back to USD.

Prerequisites

• uv — Python package manager
• just — command runner
• Docker — for building and testing the container
• Scaleway CLI (scw) — for deployment only

Quick start

1. Install dependencies

uv sync

2. Get the dataset

Download AmesHousing.csv from the EAISI discover-projects repository and place it at:

data/AmesHousing.csv

3. Train the models

just dev

This opens the Dagster UI at http://localhost:3000. Navigate to the Asset Catalog and click Materialize All. The pipeline will:

• Load and preprocess the data (outlier removal, log-transform, ordinal encoding, OHE)
• Train Ridge, LassoCV, and XGBoost
• Export models/ridge.onnx, models/lasso.onnx, models/xgboost.onnx
• Save models/preprocessing_config.json
• Evaluate the ensemble on the held-out test set

Expected test metrics: RMSLE ≈ 0.082 | R² ≈ 0.96 | MAE ≈ $11,000

4. Build the inference image

just build

Builds a multi-stage Docker image tagged mlops-demo:latest. Requires the models/ directory to be populated (step 3). The build context is the project root so both models/ and serverless/app/ are accessible in a single docker build invocation.

5. Test locally

just test

Builds the image, starts the container, runs test_predict.py against it, then stops the container. Example output:

Connecting to http://localhost:8080 ...
Health: {'status': 'ok', 'models_loaded': True}

Sending example record (AmesHousing row 1) ...

  Predicted price : $     198,432
  Actual price    : $     215,000
  Absolute error  : $      16,568  (7.7%)
  Model           : ridge+lasso+xgboost-average

Test complete.

You can also run the prediction against any live deployment:

just predict url=https://your-container.containers.fnc.fr-par.scw.cloud

6. Deploy to Scaleway

Set your Scaleway Container Registry namespace:

export SCW_REGISTRY=rg.nl-ams.scw.cloud/my-namespace
just deploy

just deploy builds the image, logs in to the registry, pushes the image, then prints the Scaleway console link. Create or update the Serverless Container there, pointing to the pushed image on port 8080.

Recipes

Recipe	Description
just dev	Launch Dagster dev server at http://localhost:3000
just build	Build Docker inference image (guards that models/ is populated)
just test	Build, start container, run smoke-test, stop container
just predict [url=...]	Post a real example record to a running container and print predicted vs. actual price
just deploy	Build, push to Scaleway registry, print console link

Project structure

mlops-demo/
├── justfile                         # Task runner recipes
├── pyproject.toml                   # uv project — pipeline dependencies
├── test_predict.py                  # Standalone smoke-test (stdlib only)
├── data/
│   └── AmesHousing.csv              # place here manually
├── models/                          # populated by Dagster at runtime
│   ├── ridge.onnx
│   ├── lasso.onnx
│   ├── xgboost.onnx
│   └── preprocessing_config.json
├── mlops_demo/
│   ├── definitions.py               # Dagster Definitions entry point
│   ├── resources/                   # DataConfig resource
│   └── assets/
│       ├── ingestion.py             # raw_housing_data
│       ├── preprocessing.py         # preprocessed_data
│       ├── training.py              # training_test_split + averaging_model
│       └── evaluation.py           # model_evaluation
└── serverless/
    ├── Dockerfile                   # multi-stage build, port 8080
    ├── requirements.txt
    └── app/
        ├── main.py                  # FastAPI app
        └── schemas.py               # Pydantic models (all 79 features)

API reference

GET /health

{ "status": "ok", "models_loaded": true }

Returns 503 if the ONNX models failed to load at startup.

POST /predict

All 79 Ames Housing features are accepted as optional fields. Missing numeric values are imputed from the training medians; missing categorical values default to "NA".

curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{"GrLivArea": 1500, "OverallQual": 7, "YearBuilt": 2003, "GarageCars": 2}'

{ "predicted_sale_price": 198432.50, "model": "ridge+lasso+xgboost-average" }

Interactive docs: http://localhost:8080/docs

Notes

• The SCW_REGISTRY variable can be set permanently at the top of the justfile instead of via export.
• The Docker image is built for linux/amd64 (required by Scaleway Serverless Containers). On Apple Silicon, just test waits 10 seconds for the emulated container to cold-start.
• Models are baked into the image at build time. Re-run just build and just deploy after each training run.
• See AGENTS.md for full implementation notes and design decisions.