Machine Learning at Scale: Managing More Than One Model in Production

Source: Towards Data Science

Before diving into scalability, feel free to read my introductory piece on the fundamentals of production‑ready ML:
Machine Learning in Production – What This Really Means

In my previous article I mentioned that I’ve spent 10 years working as an AI engineer in industry. Early on, I learned a crucial lesson:

A model in a notebook is just a mathematical hypothesis.
It only becomes valuable when its output reaches a user, powers a product, or generates revenue.

I’ve already shown you what “Machine Learning in Production” looks like for a single project.
Today, the conversation shifts to Scale: managing tens or even hundreds of ML projects simultaneously.

The evolution:
• Sandbox Era → Infrastructure Era
• Deploying a model is now a non‑negotiable skill.
• The real challenge is ensuring a massive portfolio of models works reliably and safely.

1. Leaving the Sandbox: The Strategy of Availability

To understand machine‑learning at scale, you first need to leave the “sandbox” mindset behind. In a sandbox you have static data and a single model; if it drifts, you see it, you stop it, you fix it.

When you transition to Scale Mode, you’re no longer managing a single model—you’re managing a portfolio of models. This is where the CAP theorem (Consistency, Availability, Partition Tolerance) becomes your reality.

• In a single‑model setup you can try to balance the three trade‑offs.
• At scale, it’s impossible to be perfect across all three metrics.
• You must choose your battles, and more often than not Availability becomes the top priority.

Why prioritize availability?

• With 100 models running, something is always breaking.
• If you stopped the service every time a model drifted, your product would be offline ≈ 50 % of the time.

Since we cannot stop the service, we design models to fail cleanly.

Example: Recommendation system

The user stays happy and the system stays available, even though the result is sub‑optimal. To make this work you need to know when to trigger the fallback—and that leads us to the biggest challenge at scale: monitoring.

2. The Monitoring Challenge and Why Traditional Metrics Die at Scale

When operating at scale, it’s tempting to think that ensuring a system “fails cleanly” is as simple as monitoring accuracy. In practice, accuracy alone is insufficient, and here’s why:

1. Lack of Human Consensus

• In domains like computer vision, the ground truth is usually clear (e.g., “dog” vs. “not‑dog”).
• In recommendation or ad‑ranking systems, there is no universal “gold standard.”
  • If a user doesn’t click, is the model at fault, or is the user simply not interested at that moment?

2. The Feature‑Engineering Trap

• Because we can’t measure “truth” with a single metric, we tend to over‑compensate.
• Teams often add hundreds of features, hoping that “more data” will resolve the underlying uncertainty.

3. The Theoretical Ceiling

• Teams chase marginal gains (e.g., 0.1 % accuracy improvements) without confirming whether the data is too noisy to support further progress.
• This leads to chasing an invisible performance ceiling.

---

Why This Matters

Because monitoring “truth” is nearly impossible at scale—creating dead zones where alerts never fire—we cannot rely on simple metric‑based alarms to signal failure. Consequently, we must prioritize:

• Availability – ensuring the system stays up even when the model degrades.
• Safe Fallbacks – designing mechanisms that gracefully handle “fuzzy” failures when metrics provide no clear guidance.

By focusing on these principles, we build systems that can survive ambiguous, metric‑blind failures.

3. What About the Engineering Wall

Now that we have discussed strategy and monitoring challenges, we are still not ready to scale because we haven’t addressed the infrastructure aspect. Scaling requires engineering skills just as much as data‑science skills.

We cannot talk about scaling without a solid, secure infrastructure. Because the models are complex and availability is our number‑one priority, we need to think seriously about the architecture we set up.

At this stage, my honest advice is to surround yourself with a team—or individuals—who are experienced in building large‑scale infrastructures. You don’t necessarily need a massive cluster or a supercomputer, but you do need to consider three execution basics:

• Cloud vs. Device – A server gives you power and is easy to monitor, but it’s expensive. Your choice depends entirely on the trade‑off between cost and control.
• The Hardware – You can’t put every model on a GPU; you’d go bankrupt. Adopt a tiered strategy: run simple “fallback” models on cheap CPUs and reserve expensive GPUs for the heavy “money‑maker” models.
• Optimization – At scale, a 1‑second lag in your fallback mechanism is a failure. You’re no longer just writing Python; you must compile and optimize code for specific chips so the “fail‑cleanly” switch happens in milliseconds.

4. Be Careful of Label Leakage

You’ve anticipated failures, worked on availability, sorted the monitoring, and built the infrastructure. You probably think you’re finally ready to master scalability. Not yet.
If you’ve never worked in a real environment, there’s an issue you simply can’t anticipate: label leakage. Even with perfect engineering, leakage can ruin your strategy and any system that runs multiple models.

Why It Matters

• In a single project you might spot leakage in a notebook.
• At scale—where data comes from dozens of pipelines—leakage becomes almost invisible.

The Churn Example

Imagine you’re predicting which users will cancel their subscription. Your training data includes a feature called Last_Login_Date. The model looks perfect with a 99 % F1 score.

What actually happened:

1. The database team set up a trigger that clears the Last_Login_Date field the moment a user clicks “Cancel”.
2. Your model sees a NULL login date and infers, “Aha! They canceled!”

In production, the model must make a prediction before the user cancels, i.e., before the field becomes NULL. The model is inadvertently looking at the answer from the future.

This is a simple illustration, but in complex real‑time systems (e.g., IoT), label leakage is incredibly hard to detect. The only way to avoid it is to be aware of the problem from the start.

My Tips

• Feature‑Latency Monitoring – Don’t just monitor the value of a feature; also monitor when it was written relative to the event timestamp.
• The Millisecond Test – Always ask: “At the exact moment of prediction, does this specific database row actually contain this value yet?”

These questions are simple, but the best time to evaluate them is during the design phase, before you write a single line of production code.

5. Finally, The Human Loop

The final piece of the puzzle is Accountability. At scale, our metrics are fuzzy, our infrastructure is complex, and our data is leaky, so we need a “safety net.”

Practices

• Shadow Deployment – Mandatory for scale. Deploy Model B but don’t expose its results to users. Run it “in the shadows” for a week, comparing its predictions to the eventual “truth.” If it proves stable, promote it to “live.”

• Human‑in‑the‑Loop – For high‑stakes models, keep a small team that audits the “safe defaults.” If the system falls back to “most popular items” for three consecutive days, a human must investigate why the main model hasn’t recovered.

Quick Recap Before Working with ML at Scale

1. Availability first – Because we can’t be perfect, we stay online and fail safely.
2. Fuzzy metrics – Monitoring at scale is noisy; traditional metrics are unreliable.
3. Robust infrastructure – Build cloud/hardware pipelines that make safe failures fast.
4. Guard against data leakage – Prevent “cheating” data that makes fuzzy metrics look unrealistically good.
5. Shadow Deploys – Prove a model is safe before it ever touches a customer.

Your scale is only as good as your safety net. Don’t let your work be among the 87 % of failed projects.

Connect with Me

• LinkedIn: Sabrine Bendimerad
• Medium: sabrine.bendimerad1
• Instagram: tinyurl.com/datailearn