Drafting with a small model

Inference and Serving, lesson 19

Speculative decoding: a small model drafts and the large one checks.

From "Arithmetic intensity" and "Reading a roofline chart": at small batches, a decode step costs about the same whether it handles one token or several.

Part 1. Checking is cheaper than writing

A big model at batch 1 spends its decode step reading weights. Scoring one position or five positions in the same forward pass costs about the same, just as a batch of 5 costs about the same as a batch of 1. So if someone handed the big model a guess for its next 5 tokens, it could check all of them in one step: one pass gives its own prediction at every position.

Your turn

A 70B model at batch 1 scores 5 guessed tokens in one forward pass. That costs about the same as generating how many tokens the normal way?

The step is memory-bound: its time goes on reading the weights once. Five positions share that read, the way a batch of 5 users would, and the extra maths hides inside it. So checking 5 tokens costs about one normal step.

Show the answer

B. The step is memory-bound: its time goes on reading the weights once. Five positions share that read, the way a batch of 5 users would, and the extra maths hides inside it. So checking 5 tokens costs about one normal step.

Example 1.1

One speculative step

small model→thecatsaton
A small, fast model guesses the next 4 tokens, one at a time. Each guess is cheap.
big modelone pass over all 4
wants:thecatlayonthe
The big model runs once over the prompt plus the 4 guesses, and gives its own choice at all 5 positions.
the ✓cat ✓sat ✗on
Compare left to right. The first two match. The third doesn't, so it and everything after it are thrown away.
thecatlay
Keep the matches and add the big model's own choice at the first miss. Three tokens from one big-model step. Even if every guess failed, you'd still get one.

Speculative decoding

A small draft model guesses several tokens, and the big target model checks them all in one forward pass, keeping the longest run it agrees with plus one token of its own.

Part 2. Expected tokens per step

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Expected tokens per step
Why the output doesn't change
When it helps

See Pro plans I have Pro: sign in

Next lessonPricing a single requestTurn tokens, hardware and utilisation into a cost per request. Start lesson 20

Inference and Serving overviewSpotted a mistake? Tell us.

19 · Drafting with a small model