Training and Tuning, lesson 3
Scaling laws
How loss falls with parameters, data and compute, and how to split a budget.
Before anyone trains a large model, they work out what it will cost and how big to make it. This lesson does that arithmetic: count the work, turn it into GPU-hours and money, then split a budget between model size and data. You need the training loop from "Pretraining".
By the end, you can:
Part 1. What a training run costs
Estimate a run's compute as 6 × parameters × tokens, then its GPU-hours and cost.
Every parameter does a little arithmetic for every token. In the forward pass, each weight is used in one multiply and one add: 2 operations. The backward pass does two multiply-adds for each weight, one toward its input's gradient and one toward the weight's own gradient: 4 more. So training costs about 6 operations per parameter per token.
Example 1.1
Llama 2 7B, priced out
Llama 2 7B has 7 billion parameters and read 2 trillion tokens on NVIDIA A100 GPUs, which peak at 312 trillion operations a second in the 16-bit format training uses. Say each GPU keeps up 40% of that, and a GPU-hour costs $2. What does the run take?
- Count the work6 × 7 × 10⁹ × 2 × 10¹² = 8.4 × 10²²
6 operations per parameter per token.
- Find the real speed0.4 × 312 × 10¹² ≈ 1.25 × 10¹⁴
The operations each GPU actually does in a second.
- Turn it into GPU-hours8.4 × 10²² ÷ (1.25 × 10¹⁴) ÷ 3,600 ≈ 187,000
Divide by the speed for GPU-seconds, then by 3,600. One GPU for one hour is one GPU-hour.
- Price it187,000 × $2 ≈ $374,000
Spread over 1,000 GPUs at once, the run takes about 8 days.
AnswerAbout 187,000 GPU-hours. Meta reported 184,320, within 2%.
The work is fixed by the model and the data. The speed is what engineers fight over: data has to move between chips and memory, and GPUs wait on each other. Google's PaLM paper put GPT-3's run at 21% of peak and its own at 46%. Meta reported 38% to 43% for Llama 3 405B.
FLOPs, compute and MFU
A FLOP is one floating-point operation, an add or a multiply. A run's total is its compute. MFU, model FLOPs utilisation, is the share of the hardware's peak speed a run actually gets. Careful: FLOP/s, per second, is a speed, not an amount.
compute ≈ 6 × parameters × tokens
The rule leaves out the arithmetic inside attention. The GPT-3 paper counted the same way, and put attention at under 10% of the total for its models.
Your turn
GPT-3 has 175 billion parameters and was trained on 300 billion tokens. Roughly how much compute is that?
Part 2. Twenty tokens per parameter
Split a compute budget into the best model size and number of tokens.
A budget leaves a choice: 6 × N × D can be a big model on few tokens, or a small model on many. DeepMind's Gopher had 280 billion parameters on 300 billion tokens. In 2022 they trained Chinchilla on about the same compute: 70 billion parameters on 1.4 trillion tokens. It beat Gopher on a large range of tests.
To find the best split, DeepMind trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. For each budget they asked which split gave the lowest loss. The answer: grow parameters and tokens at the same rate, at about 20 tokens per parameter.
Example 2.1
Llama 2 7B's budget, the Chinchilla way
Llama 2 7B used about 8.4 × 10²² operations. For the same budget, what model size N and token count D does the 20-to-1 rule pick?
- Put D = 20N into the cost6 × N × 20N = 120 × N²
Compute grows with the square of the model size.
- Solve for N√(8.4 × 10²² ÷ 120) ≈ 2.65 × 10¹⁰
About 26.5 billion parameters.
- Then D20 × 2.646 × 10¹⁰ ≈ 5.29 × 10¹¹
About 529 billion tokens.
AnswerAbout 26.5 billion parameters on 529 billion tokens. Meta chose 7 billion on 2 trillion, and part 4 says why.
Compute-optimal
A model trained at the best split for its budget, about 20 tokens per parameter, is compute-optimal, or Chinchilla-optimal: the lowest loss for that much training compute.
D ≈ 20 × N, so N ≈ √(compute ÷ 120)
The rule was fitted on runs of up to about 16 billion parameters and stretched to bigger ones, and it ignores everything after training. Part 4 comes back to that.
Your turn
You have 1.2 × 10²² operations to spend. Which model is compute-optimal?
Part 3. How loss falls
Read the fitted loss formula, and tell whether parameters or data help more.
DeepMind also fitted a formula that predicts the loss from N and D. It adds three parts: a floor of 1.69 that no model gets below; a model part, 406.4 ÷ N^0.34, that shrinks as the model grows; and a data part, 410.7 ÷ D^0.28, that shrinks as it reads more. N^0.34 means N to the power 0.34.
Example 3.1
Gopher and Chinchilla, through the formula
Gopher has N = 280 billion and D = 300 billion. Chinchilla has N = 70 billion and D = 1.4 trillion. Which does the formula say ends with the lower loss?
- Gopher's model part406.4 ÷ (2.8 × 10¹¹)^0.34 = 0.052
Already small, because Gopher is big.
- Gopher's data part410.7 ÷ (3 × 10¹¹)^0.28 = 0.251
Five times bigger: it read few tokens for its size.
- Chinchilla's two parts0.084 and 0.163
It gives up a little on the model part and gains a lot on the data part.
- Gopher's total1.69 + 0.052 + 0.251 = 1.993
The floor plus both parts.
- Chinchilla's total1.69 + 0.084 + 0.163 = 1.937
Lower, on about the same compute.
AnswerChinchilla, 1.937 against Gopher's 1.993.
Each doubling cuts a part by a fixed share, about a fifth, so a small part has little left to give. The cheapest gains are in whichever part is still big. Spending the same compute more evenly predicts a lower loss: the 20-to-1 rule seen from the inside.
Scaling law
A rule, fitted from many training runs, for how loss falls as you add parameters, data or compute. Chinchilla's version, with N parameters and D tokens:
loss = 1.69 + 406.4 ÷ N^0.34 + 410.7 ÷ D^0.28
Treat the fitted numbers as a sketch of the shape. In 2024, Epoch AI refitted the formula from the paper's own plots and found the published constants off; their refit agrees with about 20 tokens per parameter. The part that holds up is the balance.
Your turn
For Gopher, the model part is 0.052 and the data part 0.251. You can afford twice the parameters or twice the tokens. Which lowers the predicted loss more?
Part 4. Training past twenty
Add serving cost to training cost, and see why small models train long.
Meta trained Llama 3 8B on over 15 trillion tokens: about 1,875 per parameter, nearly 100 times the Chinchilla ratio. Meta wrote that the compute-optimal amount for an 8B model was about 200 billion tokens, and that the model kept improving all the way to 15 trillion. The reason lies after training.
Example 4.1
Training once, serving for years
Model A has 70B parameters trained on 1.4T tokens. Model B has 8B parameters trained on 15T. Running a model costs about 2 operations per parameter for each token it reads or writes. If each serves 10 trillion tokens, what does each cost in all?
- Train A6 × 70B × 1.4T = 5.9 × 10²³
A Chinchilla-style split.
- Train B6 × 8B × 15T = 7.2 × 10²³
About 22% more than A.
- Serve A2 × 70B × 10T = 1.4 × 10²⁴
Serving is forward passes only, so 2 per parameter.
- Serve B2 × 8B × 10T = 1.6 × 10²³
Almost 9 times less than A.
- Add them upA: 2.0 × 10²⁴, B: 8.8 × 10²³
B costs less than half as much overall.
AnswerA about 2.0 × 10²⁴, B about 8.8 × 10²³: less than half.
Chinchilla answers one question: the lowest loss for a training budget. It ignores the model's life after training, where every token costs 2 × N. Spend more once on a smaller model, and every request after that is cheaper. The Llama 3 paper says Meta trained its smaller models far past compute-optimal for this reason.
Overtraining
Training a model on far more than the compute-optimal 20 tokens per parameter. It costs more once, then less on every token the model serves.
lifetime ≈ 6 × N × D + 2 × N × tokens served
There's a limit. Clean text runs out, and a run that wants 50 or 100 trillion tokens has to repeat data or have other models write new text. That's why labs care so much about the work in "Building a pretraining dataset".
Your turn
Same two models: A trains for 5.88 × 10²³ and serves at 1.4 × 10¹¹ per token; B trains for 7.2 × 10²³ and serves at 1.6 × 10¹⁰ per token. After about how many served tokens does B become the cheaper one overall?
Where you'll meet it
A classic estimate: what does it cost to train a 70B model on 15 trillion tokens? Say it out loud. 6 × 70B × 15T = 6.3 × 10²⁴ operations. An H100 peaks near 989 trillion a second in 16-bit; at 40%, about 4 × 10¹⁴. That's about 4.4 million GPU-hours, near $9 million at $2 an hour.
Labs plan big runs from small ones. For Llama 3, Meta fitted scaling laws on runs from 6 × 10¹⁸ to 10²² operations and stretched them to its 3.8 × 10²⁵ budget, which suggested 402 billion parameters on 16.55 trillion tokens. It trained 405 billion on 15.6 trillion; 6 × N × D gives 3.79 × 10²⁵.
def train_flops(params, tokens):
return 6 * params * tokens
def gpu_hours(flops, peak, mfu=0.4):
return flops / (peak * mfu) / 3600
def chinchilla(flops):
params = (flops / 120) ** 0.5 # from 6 x N x 20N = 120 x N squared
return params, 20 * params
f = train_flops(7e9, 2e12) # Llama 2 7B
print(f"{f:.1e} FLOPs, {gpu_hours(f, 312e12):,.0f} A100-hours")
print(f"{gpu_hours(f, 312e12, mfu=0.2):,.0f} A100-hours at 20% MFU")
n, d = chinchilla(f)
print(f"compute-optimal: {n / 1e9:.1f}B params on {d / 1e9:.0f}B tokens")
# 8.4e+22 FLOPs, 186,966 A100-hours
# 373,932 A100-hours at 20% MFU
# compute-optimal: 26.5B params on 529B tokensChange the two numbers in train_flops to price any model whose card gives a parameter count and a token count.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A badly set-up training run keeps up 20% of its GPUs' peak speed instead of 40%. What happens to the bill?
2
Chinchilla (70 billion parameters, 1.4 trillion tokens) and Gopher (280 billion, 300 billion tokens) used about the same compute. Why did the smaller model win?
3
Training an 8B model on 15 trillion tokens costs far more than the Chinchilla split for that size. Why do it?
What to remember
1What a training run costs
Training costs about 6 × parameters × tokens operations: Llama 2 7B's 8.4 × 10²² is about 187,000 A100-hours at 40% of peak, and half the MFU doubles the bill.
2Twenty tokens per parameter
For the lowest loss on a budget, train about 20 tokens per parameter: N ≈ √(compute ÷ 120).
3How loss falls
Loss is a floor plus a model part plus a data part, so spend where the bigger part is: Chinchilla's 1.937 beat Gopher's 1.993.
4Training past twenty
Serving costs about 2 × parameters per token, so a model that will serve a lot is trained smaller and longer, like Llama 3 8B at about 1,875 tokens per parameter.
Recap and going further
The lesson on one page
- Training ≈ 6 × parameters × tokens
- 2 for the forward pass, 4 for the backward
- Divide by the speed you keep up, not the peak
- PaLM kept up 46%, Llama 3 405B 38% to 43%
- Half the MFU doubles the bill
- About 20 tokens per parameter for the lowest loss
- Best size = √(compute ÷ 120)
- Chinchilla beat Gopher on about the same compute
- A floor, a model part and a data part
- Spend where the bigger part is
- Serving costs 2 × parameters per token
- Smaller and longer is cheaper over a model's life
- Llama 3 8B: about 1,875 tokens per parameter
Try this
Price a real open model's training run
Pick two open models whose model card or paper gives both a parameter count and a training token count, for example from the Llama, Qwen, OLMo or Gemma families. For each, work out 6 × N × D, the tokens per parameter, the Chinchilla split for the same compute, and the GPU-hours at 40% of an H100's peak (about 989 trillion operations a second in BF16; NVIDIA lists 1,979 with sparsity, which doubles it). Where the paper reports real GPU-hours, compare.
You need: A spreadsheet, or the calculator code from this lesson.
Look for
- How far past 20 tokens per parameter recent small models go.
- How close your GPU-hours land to any figure the paper reports.
- What MFU the reported figure implies, if you solve for it.
- How much bigger the Chinchilla-optimal model for the same compute would be.
The words, in plain language
- FLOPs, compute and MFU
- A FLOP is one floating-point operation, an add or a multiply. A run's total is its compute. MFU, model FLOPs utilisation, is the share of the hardware's peak speed a run actually gets. Careful: FLOP/s, per second, is a speed, not an amount.
- Compute-optimal
- A model trained at the best split for its budget, about 20 tokens per parameter, is compute-optimal, or Chinchilla-optimal: the lowest loss for that much training compute.
- Scaling law
- A rule, fitted from many training runs, for how loss falls as you add parameters, data or compute. Chinchilla's version, with N parameters and D tokens:
- Overtraining
- Training a model on far more than the compute-optimal 20 tokens per parameter. It costs more once, then less on every token the model serves.
Read next
- Training Compute-Optimal Large Language ModelsJordan Hoffmann and others (DeepMind), 2022The Chinchilla paper: over 400 training runs, the 20-to-1 split and the loss formula.
- Scaling Laws for Neural Language ModelsJared Kaplan and others (OpenAI), 2020The earlier study that first showed loss falling as a smooth power law in size, data and compute.
- Chinchilla Scaling: A replication attemptTamay Besiroglu and others (Epoch AI), 2024Refits Chinchilla's formula from its own plots, finds the published constants off, and lands back near 20 tokens per parameter.
Training and Tuning overviewSpotted a mistake? Tell us.