Almost every stalled machine learning program we are called into has the same shape. There is a model. It performs well on a holdout set. Someone demoed it eight months ago and everyone agreed it was impressive. It is not in production, and nobody can give a straight answer about why.
The reason is almost never the model. It is that the distance between a notebook and a production system is an engineering problem nobody was staffed to solve, and it has a predictable structure. If you know the structure, you can plan for it instead of discovering it one painful surprise at a time.
The four gaps
In practice the distance breaks down into four separate problems that get bundled together and called 'productionisation', which is why they never get scheduled properly.
- Reproducibility. Can you rebuild the exact model that is currently serving traffic, from the exact data it was trained on? If the answer involves someone's laptop, you do not have a model - you have an artifact nobody can regenerate.
- Serving. The model needs to run somewhere with a latency budget, a scaling story, and a cost ceiling. A model that takes four seconds is a different product from one that takes forty milliseconds, and that difference is usually discovered late.
- Evaluation. Not the offline metric from the notebook, but an automated gate that runs on every change and blocks a release that regresses. Without it, every deployment is a judgment call made under time pressure.
- Monitoring. The world moves. Inputs drift, upstream schemas change, a feature silently starts arriving null. You need to find out from a dashboard, not from a customer.
Each of these is tractable on its own. Together they represent a platform, and platforms need to be someone's job. The most common organisational failure we see is a data science team that is measured on model quality being implicitly expected to also deliver all four, with no platform engineering support and no time allocated for it.
Reproducibility is the one to fix first
If you can only address one gap this quarter, make it reproducibility, because the other three depend on it. You cannot meaningfully evaluate a model you cannot rebuild. You cannot roll back to a previous version you cannot reconstruct. You cannot investigate a drift alert without knowing exactly what the model was trained on.
Concretely, this means a registry that records, for every model version: the training data snapshot, the code commit, the hyperparameters, the environment, and the evaluation results. Nothing exotic - this is a solved problem with several good open-source options. The hard part is the discipline of routing every model through it, including the urgent one that someone wants to ship on Friday.
A model you cannot rebuild is not an asset. It is a dependency with an unknown expiry date.
Evaluation gates change team behaviour
The second-highest-leverage change is putting an automated evaluation gate in your deployment pipeline. Not a report someone reads - a check that fails the build.
The effect on team behaviour is larger than the technical effect. Once a regression blocks a merge, model quality stops being a matter of opinion in a review meeting and becomes a property of the pipeline. Arguments about whether a change is an improvement get resolved by running it. Teams that adopt this consistently report that their release frequency goes up, not down, because the risk of each individual release drops.
Design the gate to test what you actually care about. An aggregate accuracy threshold is a weak gate - it hides regressions in the subgroups that matter most. Slice your evaluation set along the dimensions where a failure would be expensive and gate on the worst slice, not the average.
Monitoring: watch inputs, not just outputs
Most teams that instrument anything instrument prediction distributions. That is worth doing, but it is a lagging indicator - by the time output distribution has visibly shifted, the model has been making worse decisions for a while.
Input monitoring catches problems earlier and diagnoses them faster. Track feature distributions, null rates, and cardinality on the live inference path. The overwhelming majority of real production incidents we have investigated were upstream data problems, not model degradation: a schema change nobody announced, a join that started silently dropping rows, a unit change in a source system.
What a reasonable first quarter looks like
For a team with models in production and no platform, a realistic first quarter is: a registry with every current production model registered and reproducible, one model moved onto an automated deployment path with an evaluation gate, and input monitoring on that model's live traffic. That is genuinely achievable in twelve weeks, and it establishes the pattern the rest of your models follow.
What does not work is attempting all four gaps across every model simultaneously. That is a platform rewrite disguised as a quarter of work, and it tends to end with a partially migrated estate that is harder to operate than what you started with.
