insideout Get Pro

06 · When to page someone

Pro

Production AI, lesson 6

When to page someone

Alerts worth waking up for, and how to run an incident.

"Logs, metrics and traces" covered what to record. This lesson is about which of those numbers should wake someone up, and what to do when they do.

Part 1. Page on what users feel

It's 3am and your phone buzzes: "GPU memory at 92% on inference node 4". You get up, look, and find every customer is being served fine. The next week nothing buzzes at all while a third of answers come back empty for two hours, because nobody wrote an alert for that.

Both mistakes come from alerting on causes (a busy machine, a full queue) instead of symptoms (customers getting errors, slow answers or bad answers). Causes are many and often harmless. Symptoms are few, and they always matter.

Your turn

Which of these should wake an engineer at 3am?

Only that one is customers being hurt right now. The busy CPU and the degraded provider might lead to harm, so put them on a dashboard, not a pager. The spend creep is real but can wait for a ticket in the morning.

Symptom-based alerting

Paging on what users experience (errors, latency, broken answers) rather than on internal causes. Cause signals still matter, for finding the problem once a symptom alert has fired.

Part 2. SLOs and error budgets

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. SLOs and error budgets
  2. Burn rate: how fast you're spending
  3. LLM symptoms, and jobs that fail quietly
  4. Running the incident
See Pro plansI have Pro: sign in
Next lessonDrift and model monitoringNotice when inputs or quality shift after launch. Start lesson 7

Production AI overviewSpotted a mistake? Tell us.