From Notebook to Production: What a Machine Learning System Actually Contains

Quick take. The model is the smallest part of a production machine learning system. Prototypes stall because the rest (data checks, a shared feature pipeline, serving, monitoring, rollback and an owner) was never built. Treat the notebook as a specification, and score readiness before you launch.

The model is the small box

In 2015, Sculley and colleagues at Google published a diagram worth keeping on every machine learning team’s wall [1]. A tiny black box labelled “ML Code” sits in the middle. Around it sit much larger boxes: configuration, data collection, data verification, feature extraction, machine resource management, analysis tools, process management tools, serving infrastructure and monitoring. The caption says that only a small fraction of real-world machine learning systems is composed of machine learning code [1]. Elsewhere, the authors note that a mature system might end up as at most 5% machine learning code and at least 95% glue code [1].

They also name a “prototype smell”. Relying on a prototyping environment signals a brittle full-scale system, and time pressure can push a prototype into service as the production solution [1].

Carrying models beyond the prototype and into production, where they support the decisions operations teams make every day, is the part of the work I focus on. Here is what that stretch involves.

Why prototypes stall

Paleyes, Urma and Lawrence surveyed published deployment case studies and found that practitioners face challenges at every stage of the deployment workflow: data management, model learning, model verification and model deployment [2]. In practice, five gaps come up again and again:

1. The data was a snapshot. The notebook used a clean extract. Production data arrives late, with renamed sensor tags, changed units and missing columns.
2. Features are computed twice. Training features come from historical tables; serving features come from a separate real-time code path. The two drift apart, a problem known as training/serving skew [3].
3. The offline metric is not the business metric. A better validation score does not mean better outcomes unless someone has checked that the two move together [3].
4. There is no way back. Without a canary release or a rollback path, every deployment is a gamble.
5. No one owns it. The person who built the model moves on, and nobody is alerted when it misbehaves.

Amershi and colleagues at Microsoft found three reasons machine learning differs from earlier software work: data is harder to discover, manage and version than code; customizing and reusing models takes different skills; and models are harder to isolate as modules, because they can be “entangled” with each other [4].

What the full system contains

This list lines up with the literature. Kreuzberger, Kühl and Hirschl combined a literature review, a tool review and expert interviews to define nine principles of machine learning operations (MLOps): automation of continuous integration and delivery, workflow orchestration, reproducibility, versioning of data, code and model, collaboration, continuous training and evaluation, metadata tracking, continuous monitoring and feedback loops [5]. Amershi and colleagues describe a nine-stage workflow, from model requirements through data collection, cleaning and labelling, feature engineering, training and evaluation to deployment and monitoring, with feedback loops from evaluation and monitoring back to earlier stages [4].

Score readiness before launch

Breck and colleagues turned production readiness into a number, the ML Test Score [3]. It has 28 tests in four sections of seven: data, model development, machine learning infrastructure and monitoring. Scoring works like this [3]:

• half a point for each test run manually, with the results documented and shared;
• a full point for each test that runs automatically on a repeated basis;
• add up each section separately;
• the final score is the minimum of the four section scores.

The minimum is the important design choice: a system is only as ready as its weakest area. Here is an illustrative team.

Final score = min(4.5, 3.0, 1.5, 0.5) = 0.5. This team did solid data work, but nobody is watching the model after launch, and the score says so.

These tests are not universal habits, even at large companies. In the authors’ survey of several dozen teams at Google, none of the tests was implemented by more than 80% of the teams [3].

Treat the notebook as a specification

The notebook proves the idea. It should not become the product. Turn it into a specification, then build the system around it:

1. Freeze what the notebook established: target definition, label window, feature list, evaluation metric and the baseline the model must beat.
2. Rewrite the feature code once, as a shared library that both training and serving call. Then test for skew by logging serving inputs and comparing them with features recomputed by the training path [3].
3. Version data, code and model together, so any prediction can be traced to the exact model and data behind it [5].
4. Run in shadow mode first: score live data, log the predictions, take no action, and compare against what actually happened.
5. Release with a canary that sends a small share of traffic to the new model, and practise rollback before you need it [3].
6. Name an owner, an alert route and a retraining trigger before go-live.

Shankar and colleagues interviewed 18 machine learning engineers and identified three variables that govern success in production: velocity, validation and versioning [6]. The steps above serve all three.

Launch checklist

• Input data is validated against a schema at training and at serving time.
• Training and serving share one feature code path, and skew is measured.
• Training is reproducible from versioned data and code.
• The model beats a simple baseline on the decision metric, including on important data slices [3].
• Serving meets its latency and throughput budget.
• Inputs, predictions, outcomes and model age are monitored, with alert thresholds.
• Canary release and rollback have been tested.
• An owner and a retraining trigger are named.

A prototype answers the question “can this work?” A production system answers a harder one: “will this keep working next month, and will we know if it stops?” Build for the second question from the first sprint.

References

1. D. Sculley et al., “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015, pp. 2503-2511.
2. A. Paleyes, R.-G. Urma, and N. D. Lawrence, “Challenges in deploying machine learning: A survey of case studies,” ACM Computing Surveys, 2022.
3. E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ML test score: A rubric for ML production readiness and technical debt reduction,” in Proc. IEEE Int. Conf. Big Data, 2017.
4. S. Amershi et al., “Software engineering for machine learning: A case study,” in Proc. IEEE/ACM 41st Int. Conf. Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019.
5. D. Kreuzberger, N. Kühl, and S. Hirschl, “Machine learning operations (MLOps): Overview, definition, and architecture,” IEEE Access, vol. 11, pp. 31866-31879, 2023.
6. S. Shankar, R. Garcia, J. M. Hellerstein, and A. G. Parameswaran, “Operationalizing machine learning: An interview study,” arXiv:2209.09125, 2022.