insideout Get Pro

18 · Prefix caching

Pro

Inference and Serving, lesson 18

Prefix caching

Reuse the work for a shared system prompt or document.

You know that prefill builds keys and values for every prompt token ("Reading the prompt, writing the reply", "Caching keys and values"), and that servers store the cache in 16-token blocks ("How big the cache gets").

Part 1. The same prompt, again and again

A support bot sends a 3,000-token system prompt and tool list with every message. The user's question adds 50 tokens. So every request, prefill computes keys and values for the same 3,000 tokens it computed for the request before. That's 98% of the prefill work, repeated.

Your turn

Two requests share the same 3,000-token start and then ask different questions. Are the keys and values for those first 3,000 tokens the same in both?

In a decoder, token 2,000's keys and values depend only on tokens 1 to 2,000. Whatever comes after can't reach back and change them. So the KV cache for a shared start can be saved once and reused by any request that begins the same way. Sampling settings only affect what gets generated later.

Prefix caching

Holding on to the KV cache built for the start of a prompt after its request is done, so a later request that begins with exactly the same tokens skips prefill for that part. Prefill then runs only on what's new.

Prefix caching saves compute and cuts time to first token, since prefill on 50 tokens is far quicker than on 3,050. It costs GPU memory to hold the saved caches, so servers drop the least recently used ones when space runs out.

Part 2. One different token, and everything after it misses

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. One different token, and everything after it misses
  2. A tree of shared prefixes
  3. Prompt caching in an API
See Pro plansI have Pro: sign in
Next lessonDrafting with a small modelSpeculative decoding: a small model drafts and the large one checks. Start lesson 19

Inference and Serving overviewSpotted a mistake? Tell us.