insideout Get Pro

06 · Percentiles and the slow tail

Pro

Inference and Serving, lesson 6

Percentiles and the slow tail

Why p99 matters more than the average once traffic is real.

You know how to time one request's first token and the gaps after it ("First token and every token after"). Now you measure many.

Part 1. The average lies

Send the same request 20 times and you get 20 different times. Here are 20 realistic end-to-end times in seconds, sorted: 0.82, 0.85, 0.88, 0.90, 0.91, 0.93, 0.95, 0.97, 0.98, 1.00, 1.02, 1.04, 1.05, 1.08, 1.10, 1.15, 1.20, 1.30, 2.40, 9.60.

Your turn

The average of those 20 is about 1.51 seconds. How many of the 20 requests actually took about 1.5 seconds?

Eighteen requests landed between 0.82 and 1.30 s. Two landed far out, at 2.40 and 9.60. The 9.6 s request alone pulls the average up by about 0.4 s. The average describes a request that never happened.

Latency is almost never spread evenly around its average. Most requests bunch up near a typical value, and a few trail off far to the slow side. Those few are where users give up.

Part 2. Percentiles

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Percentiles
  2. Why the tail matters more than it looks
  3. Percentiles across a fleet
See Pro plansI have Pro: sign in
Next lessonPrompt length on different GPUsHow prompt length and hardware set prefill time. Start lesson 7

Inference and Serving overviewSpotted a mistake? Tell us.