Inference and Serving, lesson 1
Reading the prompt, writing the reply
Prefill reads the whole prompt, then decode writes one token at a time.
Every reply from a language model is made in two phases, and they spend time in very different ways. This lesson splits a request into those phases and works out by hand how long each takes on a real GPU, and what it costs. You need to know that a model writes one token at a time and feeds each one back in ("Autoregressive generation" in Transformers).
By the end, you can:
- Split a request into prefill and decode, and count the passes each needs.
- Count operations per byte for each phase, and tell which limit sets its pace.
- Estimate the time to the first token and the tokens per second from model size and GPU specs.
- Work out how batching users together cuts the cost per million tokens.
Part 1. The prompt is known, the reply isn't
Split a request into prefill and decode, and count the passes each needs.
You send a 2,000-token question and get a 300-token answer. Every prompt token exists before the model starts. The answer's tokens don't: the second can't be chosen until the first has been. So the model can't treat the two the same way.
Example 1.1
One request, counted in passes
A request has a 2,000-token prompt and a 300-token reply. How many forward passes (runs through every layer of the model) does it take, and how many tokens does each pass carry?
- Read the prompt1 pass × 2,000 tokens
All the prompt tokens are known, so they go through the layers side by side, as in training. The causal mask keeps each one looking back only.
- Take the first reply token1 token out
The scores at the last prompt position give reply token 1. The first pass ends here.
- Write the rest300 − 1 = 299 passes
Each later token needs the one before it as input, so each gets a pass of its own, carrying that one token.
- Add them up1 + 299 = 300 passes
One pass per reply token. The prompt's length sets how big the first pass is, not how many passes there are.
Answer300 passes: one carrying 2,000 tokens, then 299 carrying one token each.
Why can't the reply go through in one pass too? Because reply token 2 is an input to the pass that picks token 3. The prompt has no such chain. Every prompt token is known before the model starts, so the work on all of them can happen at once.
Prefill and decode
Prefill is the first pass: it reads the whole prompt at once and produces the first reply token. Decode is everything after: one token per pass, each pass taking the token just made as its new input.
passes = 1 + (reply tokens − 1)
A long prompt makes prefill bigger but adds no passes. A long reply adds one pass per token. So summarising a 20,000-token report into 100 tokens is mostly prefill, and writing a 2,000-token story from a one-line prompt is mostly decode.
Your turn
A request has a 500-token prompt and a 1,000-token reply. How many forward passes does it take?
Part 2. Same weights, very different work
Count operations per byte for each phase, and tell which limit sets its pace.
A GPU keeps the model's weights in its main memory. Its arithmetic units can only use a weight once it has been read in from there, and every pass reads all of them, whether it carries one token or thousands. Once read, each weight does about 2 operations for every token in the pass: a multiply and an add.
Example 2.1
Llama 3 8B, both phases
Llama 3 8B has 8.03 billion weights, stored in 16 bits (2 bytes) each. For a 3,000-token prompt, how many operations does prefill do for each byte of weights it reads? And one decode step?
- Size the weights8.03 B × 2 = 16.06 GB
Every pass reads all 16.06 GB.
- Count prefill's operations2 × 8.03 B × 3,000 = 48.2 T
2 operations per weight for each of 3,000 tokens: 48.2 trillion.
- Divide by the bytes48.2 T ÷ 16.06 G ≈ 3,000
Prefill does about 3,000 operations for every byte it reads.
- Do the same for decode16.06 G ÷ 16.06 G = 1
One token: 2 × 8.03 billion operations on the same 16.06 GB.
AnswerAbout 3,000 operations per byte for prefill, and 1 for a decode step.
With 16-bit weights the answer is just the number of tokens in the pass, since each 2-byte weight does 2 operations per token. Now the GPU. An H100 SXM can read 3.35 trillion bytes a second from its memory, and do about 990 trillion 16-bit operations a second. That's about 295 operations in the time it reads one byte.
Compute-bound and memory-bound
A pass is compute-bound when its arithmetic takes longer than reading its data, like prefill. It's memory-bound when reading takes longer, like decode. To tell which, compare its operations per byte with the GPU's own ratio.
16-bit weights: operations per byte ≈ tokens in the pass
Two edges. Short prompts are memory-bound too: below about 295 tokens, prefill waits on memory just as decode does. And NVIDIA's spec sheet lists 1,979 trillion operations a second, but that figure counts a sparsity trick most models don't use. Ordinary dense maths gets half.
Your turn
The same model on the same GPU gets a 200-token prompt. About how many operations per byte does its prefill do, and what sets its pace?
Part 3. Time on the clock
Estimate the time to the first token and the tokens per second from model size and GPU specs.
Those two limits turn straight into time. Prefill is compute-bound, so its time is its operations divided by the GPU's operations a second. A decode step is memory-bound, so its time is the bytes it reads divided by the bytes a second the memory delivers. Each step makes one token.
Example 3.1
Llama 3 8B on one H100
Same model and 3,000-token prompt, on one H100 (990 trillion operations and 3.35 trillion bytes a second). The reply is 250 tokens. At best, how long until the first token, how fast do the rest come, and how long does the whole reply take?
- Time the prefill48.2 T ÷ 990 T/s ≈ 49 ms
Operations divided by operations a second.
- Time one decode step16.06 GB ÷ 3.35 TB/s = 4.79 ms
Bytes divided by bytes a second.
- Turn it into a rate1 s ÷ 4.79 ms ≈ 209
One token per step: about 209 tokens a second.
- Time the whole reply49 ms + 249 × 4.79 ms ≈ 1.24 s
The prefill, then 249 decode steps for the other 249 tokens.
AnswerThe first token after about 49 ms, then about 209 tokens a second: 1.24 s in all, 96% of it decode.
Prefill did twelve times the arithmetic of the 249 decode steps, yet took less than a twentieth of the time. Decode pays for reading all 16 GB of weights again for every single token. That read, not the maths, is what makes a reply slow.
TTFT and TPOT
The wait before a reply starts is the time to first token (TTFT), mostly prefill. The gap between later tokens is the time per output token (TPOT), one decode step. With one user on one GPU, a step can't beat the weight bytes divided by the memory bandwidth, the bytes a second the GPU can read:
time per token ≥ weight bytes ÷ memory bandwidth
These are best cases. No real program runs at the spec sheet's peak, so measured times are longer. A real server also adds network and queueing time before prefill even starts, which "First token and every token after" takes apart.
Your turn
You store Llama 3 8B's weights in 8 bits (1 byte) each instead of 16. On the same H100, what's the most tokens a second one reply can get?
Part 4. Many users, one read of the weights
Work out how batching users together cuts the cost per million tokens.
With one user, the H100 spends nearly all of each decode step reading weights while its arithmetic units sit idle. A server fills that idle time by putting many users' decode steps into the same pass. The weights are read once, and every token in the pass uses them.
Example 4.1
The cost of a million tokens
Say the H100 rents for $3 an hour. What does a million output tokens cost with one user per pass, and with 32 users per pass? Leave out each user's stored state for now: the next two lessons add it.
- Tokens an hour, one user209 × 3,600 ≈ 750,000
One token every 4.79 ms.
- Cost per million, one user$3 ÷ 0.75 = $4.00
An hour costs $3 and makes 0.75 million tokens.
- Tokens an hour, 32 users32 × 750,000 = 24 million
Each step still reads 16.06 GB and takes about 4.79 ms, but now makes 32 tokens.
- Cost per million, 32 users$3 ÷ 24 = $0.125
32 times cheaper, and each user's reply comes just as fast.
Answer$4.00 per million tokens with one user, $0.125 with 32.
Batching works because decode is memory-bound. A pass carrying 32 tokens does 32 operations per byte, far under the H100's 295, so it takes about as long as a pass carrying one. Somewhere past 295 tokens a pass, arithmetic becomes the limit, and each extra user starts to slow every step.
Batching
Putting several requests into one forward pass, so they all share a single read of the weights. For memory-bound decode it multiplies the tokens a second at almost no cost in speed for each user.
cost per million tokens = price an hour ÷ millions of tokens an hour
The limit is memory. Each user in the batch brings state of their own, kept on the same GPU, and the next two lessons work out how big it gets.
Your turn
Same GPU at $3 an hour, now with 8 users per pass. About what does a million output tokens cost?
Where you'll meet it
A common question: why does a model write so much more slowly than it reads, and how would you speed it up? Decode reads every weight for every token, so memory bandwidth sets its pace: 16 GB at 3.35 TB/s is about 4.8 ms a token. The fixes: batch more users into each pass, or store the weights in fewer bits.
Serving engines report the two phases separately. Ollama's --verbose mode prints a prompt eval rate and an eval rate, and vLLM logs prompt and generation throughput. A slow start points at prompt length. A slow trickle points at decode: the model's size in bytes, and how many users share each pass.
params = 8.03e9 # Llama 3 8B
weight_bytes = params * 2 # 16-bit weights
flops, bandwidth = 990e12, 3.35e12 # H100 SXM, per second
prompt, reply = 3000, 250
first = 2 * params * prompt / flops # compute-bound
step = weight_bytes / bandwidth # memory-bound
total = first + (reply - 1) * step
print(f"first token {first * 1000:.0f} ms, then {1 / step:.0f} tokens/s")
print(f"whole reply {total:.2f} s, {(total - first) / total:.0%} decode")
for users in (1, 8, 32):
per_hour = users * 3600 / step
print(f"batch {users:>2}: ${3 / (per_hour / 1e6):.3f} per million tokens")
# first token 49 ms, then 209 tokens/s
# whole reply 1.24 s, 96% decode
# batch 1: $3.995 per million tokens
# batch 8: $0.499 per million tokens
# batch 32: $0.125 per million tokensChange the GPU's numbers, the weight size or the batch to see how each one moves the time and the cost.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
Which request spends the largest share of its time in decode?
2
Model B has twice as many weights as model A, stored the same way, on the same GPU. With one user, how does B's decode speed compare?
3
Why can't decode handle ten reply tokens in one pass, the way prefill handles ten prompt tokens?
4
Decode is memory-bound. You run 8 users' decode steps together in one pass instead of one at a time. What happens to the weight reads per token?
What to remember
1The prompt is known, the reply isn't
Prefill reads the whole prompt in one pass, then decode makes one token per pass, so a 300-token reply takes 300 passes.
2Same weights, very different work
With 16-bit weights, a pass does about one operation per byte for each token it carries: 3,000 for a 3,000-token prefill and 1 for a decode step, against the H100's 295.
3Time on the clock
Time per token is at least weight bytes ÷ memory bandwidth: 16.06 GB ÷ 3.35 TB/s = 4.79 ms, about 209 tokens a second.
4Many users, one read of the weights
Batching shares each read of the weights: 32 users per pass cut $4.00 per million tokens to $0.125.
Recap and going further
The lesson on one page
- Reads the whole prompt in one pass
- Ends with the first reply token
- One pass per reply token
- Each pass waits on the one before
- 16-bit weights: about the tokens in the pass
- An H100 needs about 295 to keep busy
- First token: operations ÷ operations a second
- Each later token: weight bytes ÷ bandwidth
- Batching shares each read of the weights
- 32 users: $4.00 down to $0.125 per million tokens
Try this
See the two phases on your own laptop
Install Ollama and run a small model with timing turned on: ollama run llama3.2:1b --verbose. Ask it something short. Then paste in a long article (a few thousand tokens) and ask for a one-line summary. Then ask for a 500-word story. After each reply, Ollama prints a "prompt eval rate" (prefill) and an "eval rate" (decode), both in tokens per second. Last, divide your machine's memory bandwidth (from its spec sheet) by the model's size in ollama list.
You need: A laptop with Ollama installed from ollama.com. A 1B model runs without a GPU.
Look for
- How many times faster the prompt eval rate is than the eval rate.
- Which of the three requests spends most of its time in prompt eval, and which in eval.
- How close the eval rate gets to your bandwidth ÷ model size ceiling.
The words, in plain language
- Prefill and decode
- Prefill is the first pass: it reads the whole prompt at once and produces the first reply token. Decode is everything after: one token per pass, each pass taking the token just made as its new input.
- Compute-bound and memory-bound
- A pass is compute-bound when its arithmetic takes longer than reading its data, like prefill. It's memory-bound when reading takes longer, like decode. To tell which, compare its operations per byte with the GPU's own ratio.
- TTFT and TPOT
- The wait before a reply starts is the time to first token (TTFT), mostly prefill. The gap between later tokens is the time per output token (TPOT), one decode step. With one user on one GPU, a step can't beat the weight bytes divided by the memory bandwidth, the bytes a second the GPU can read:
- Batching
- Putting several requests into one forward pass, so they all share a single read of the weights. For memory-bound decode it multiplies the tokens a second at almost no cost in speed for each user.
Read next
- Transformer Inference ArithmeticCarol Chen (kipply)Counts the bytes and operations of prefill and decode by hand, the way this lesson does.
- LLM Inference Performance Engineering: Best PracticesDatabricks blogThe latency terms this lesson uses, and how close real models on real GPUs get to their memory bandwidth.
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model ServingZhong and colleagues, 2024Measures how the two phases interfere and serves them on separate GPUs.
Inference and Serving overviewSpotted a mistake? Tell us.