insideout Get Pro

12 · Canary, shadow and rollback

Pro

Production AI, lesson 12

Canary, shadow and rollback

Release a change to a few users first, and undo it fast.

"Logs, metrics and traces" and "When to page someone" covered how you see a problem. This lesson is about shipping changes so that a problem reaches as few people as possible.

Part 1. A prompt change is a deploy

Someone adds "Be concise." to the support assistant's system prompt. Answers get shorter. They also stop including the order number, which another system was reading out of the text. Nothing in the code changed, and the refunds team spends the afternoon matching orders by hand.

In an LLM app, the behaviour lives in more than code: the model snapshot, the prompt, retrieval settings, guardrail thresholds, tool descriptions. A change to any of them is a release, and should get the same care: a version number, a review, a test run, and a way back.

Your turn

Your code and prompts didn't change, but answer quality dropped overnight. What's a likely cause?

Many providers offer names like "latest" that move when a new version ships. Your app changed under you without a deploy. Pin exact model snapshots in production, and move to a new one on purpose, through the same steps as any other release.

So a prompt change and a model upgrade go through the same steps: a version number, an eval run against the fixed set, a shadow sample of real traffic, a canary that widens in stages, and a flag that turns it off. Skipping steps is fine for a typo fix. It's a gamble for a new model.

Some releases are trained models, like the small classifier in the next lesson. For those, a way back needs records. Every training run should save its code version, its exact data, its settings and its scores, and the model you ship gets a version number tied to that run.

Experiment tracking and model registry

Experiment tracking records every training run (code, data, settings, scores) so runs can be compared and rebuilt. A model registry lists the models you've released by version, each linked to the run that made it, and marks which one is live. In MLflow, serving can load whichever version holds an alias you control, such as champion, so rolling back means moving it to the old version.

Your turn

A retrained classifier, version 8, misbehaves, so you move the live alias back to version 7. Errors stay high. Version 8's release also changed how the service computes one of the model's inputs. What's wrong?

A model and the code that prepares its inputs ship together, so roll them back together, or keep the input code working for every version you might return to. "Feature stores and training-serving skew", later in this module, covers this kind of mismatch.

Part 2. Shadow mode: try it on real traffic, show no one

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Shadow mode: try it on real traffic, show no one
  2. Canary releases and automatic rollback
  3. Kill switches and A/B tests
See Pro plansI have Pro: sign in
Next lessonCheap decisions before expensive modelsA small classifier decides first, and the large model only writes. Start lesson 13

Production AI overviewSpotted a mistake? Tell us.