The application is deliberately boring: type a restaurant review, get “positive” or “negative”, tell the system whether it was right. The point of the project was never the model. It was everything around the model, which is where machine learning systems actually break.

This was the team project for Release Engineering for Machine Learning Applications at TU Delft, built over a quarter by a team of five across seven repositories.

Seven repositories, one system

The split is the first design decision, and it drives everything else. A single repository would have been simpler to work in and would have hidden every problem the course was about.

lib-ml exists because of the single most common way ML systems go wrong in production: training and serving preprocess text differently, and nobody notices until the accuracy quietly drops. Making preprocessing a versioned package that both sides import removes the possibility.

Every repository has its own release workflow. Services use bumpver with -alpha pre-release tags, and pushing a vX.Y.Z tag triggers a GitHub Actions workflow that builds and publishes. Promoting an alpha to a stable release is a deliberate manual act, not a side effect of merging.

The training pipeline

A GaussianNB classifier over bag-of-words features, wrapped in a four-stage DVC pipeline: get_data, preprocess, train_model, evaluate. Data and model artefacts live on a Google Drive DVC remote accessed through a service account, so dvc repro reproduces a run from a clean checkout and dvc exp run -S train.random_state=45 sweeps a parameter without polluting git history.

Held-out performance:

The gap between 0.986 training and 0.711 test accuracy is real overfitting, and we left it visible rather than tuning it away. The pipeline is the deliverable; the numbers it reports are just what this model does.

Testing an ML pipeline

The test suite follows Google’s ML Test Score methodology, which scores a project on five axes rather than on line coverage alone. Our scripts/ml_test_score.py parses the JUnit XML and weights the categories: data and feature integrity (30), model development (25), ML infrastructure (20), monitoring (15), metamorphic testing (10).

The badges the CI writes back into the README:

The metamorphic tests are the interesting ones. A metamorphic test asserts a relation between outputs rather than a specific output, which is how you test a model whose correct answer you do not know. Ours swaps two randomly chosen bag-of-words features in a test sample and asserts the prediction is unchanged, under a fixed seed so failures reproduce.

For linting we run pylint with a custom linters module that flags ML-specific code smells rather than generic Python ones:

• Uncontrolled randomness: any operation that samples without a seed set.
• Implicitly set hyperparameters: relying on a library default instead of naming the value.
• Hardcoded dataset paths: absolute paths that will not survive the container.

Plus flake8 for style and complexity limits, bandit for static security scanning (it catches the pickle loads, which we then explicitly justify), and radon for cyclomatic complexity.

The cluster

Everything runs on a Kubernetes cluster provisioned from scratch on the developer’s own machine: ./scripts/run-all.sh brings up the VMs, runs the Ansible playbooks, installs the Helm charts, and waits for pods to become ready. Provisioning is idempotent, so a timeout is not a disaster, and downloaded models are cached in a shared folder mounted into every VM so the second run is fast.

The deployment is split into two Helm charts on purpose. app-chart holds the serving path: the three microservice deployments, the Istio gateway, and the routing rules. monitoring-chart holds Prometheus, Grafana, and Kiali behind a separate gateway. Monitoring evolves independently of what it monitors, and you can install a second application release without duplicating the observability stack.

Routing

Every pod gets an injected Envoy sidecar, which is what makes the traffic control possible without touching application code. Three mechanisms, layered:

Weighted split. The frontend VirtualService sends 90% of traffic to v1 and 10% to v2. The weights are Helm values and can be shifted at runtime, which is the entire canary-rollout mechanism and also the rollback mechanism.

Header-based override. A request carrying user-group: canary always reaches frontend v2, regardless of weights. Separately, a label: v1 or label: v2 header pins both app-service and model-service to a matching version, so a request cannot be served by a v2 backend talking to a v1 model. Without the header those two split 50/50.

Sticky sessions. Istio’s load balancer uses consistent hashing on a custom x-user header, so a given user stays on the same version for the duration of their session. Without this, a 90/10 split means a user flips between UIs mid-session, which poisons any experiment you try to run.

Monitoring

Four custom application metrics, on top of what Istio and Prometheus provide by default:

A Grafana dashboard is shipped as a ConfigMap inside the Helm chart and auto-provisioned, so it appears on install with no manual import step. There is one custom PrometheusRule, TooManyActiveUsers, which fires above a configurable threshold and routes to Alertmanager for email delivery.

Continuous experimentation

The experiment we ran on this infrastructure: does replacing the “Correct” and “Incorrect” feedback buttons with thumbs-up and thumbs-down icons increase the fraction of predictions that get feedback?

The versions are v1 (buttons) and v2 (icons), deployed simultaneously behind a 90/10 split, with sticky sessions keeping each user on one side. The metric is the feedback-to-prediction ratio, computed per frontend version in PromQL:

sum by (frontend_version) (
  label_replace(increase(frontend_feedback_rating_total[1h]),
    "frontend_version", "$1", "pod",
    "myapp-app-frontend-(v[0-9]+)-deployment.*")
)
/
sum by (frontend_version) (
  label_replace(increase(frontend_prediction_requests_total[1h]),
    "frontend_version", "$1", "pod",
    "myapp-app-frontend-(v[0-9]+)-deployment.*")
)

The label_replace is doing the real work: pod names carry the version, but there is no version label on the metric itself, so the regex lifts it out of the pod name into a label you can group by.

The honest caveat: the application was never deployed publicly and we ran no study with real participants, so the traffic behind these panels is generated, not observed. What the experiment demonstrates is that the measurement apparatus works end to end, from a feature branch through a tagged pre-release image, into a weighted Istio route, out to a version-labelled Prometheus query. The decision procedure was written down in advance (a statistically significant increase of at least 10% in feedback rate, then a gradual rollout with regression monitoring) but was never exercised against real data.

What I would change

The gap this system still has is the one we wrote up as the extension proposal: the feedback the frontend so carefully collects goes into a Prometheus counter and nowhere else. The labels are thrown away. A model trained once and deployed forever drifts, and we were sitting on a stream of free labels without using them.

The proposal closes that loop: feedback events streamed through Kafka into a labelled store, a curation step to deduplicate and score them, a scheduled Kubeflow pipeline that retrains, and validation gates that require the new model to beat the incumbent on held-out metrics before it is allowed into the existing 10% canary slot. The rollout machinery to support that already exists in this project, which is the point. Only the training trigger and the data path are missing.