Training and Tuning, lesson 1
Building a pretraining dataset
Where training text comes from, and how it's cleaned and mixed.
A model can only learn what's in its training text, and most of that text starts as messy web pages. This lesson follows web text from the crawl to the final mix, counting tokens on the way. You need to know what a token is ("Text as bytes, bytes as tokens" in Tokens and Embeddings).
By the end, you can:
Part 1. Counting text in tokens
Size a pile of text in tokens from its character count.
Training text is counted in tokens. Meta trained the largest Llama 3 model on 15.6 trillion of them. A pile that size comes mostly from the public web, through Common Crawl, a non-profit that has crawled the web since 2007. It adds 3 to 5 billion pages a month and gives everything away.
Example 1.1
A month of the web, in tokens
Say one month's crawl holds 3 billion pages, and the main text of a page averages 2,000 characters. English runs about 4 characters to a token. How many tokens is the crawl, and how many crawls like it would make 15.6 trillion tokens?
- Count the characters3 × 10⁹ × 2,000 = 6 × 10¹²
3 billion pages of 2,000 characters: 6 trillion characters.
- Turn them into tokens6 × 10¹² ÷ 4 = 1.5 × 10¹²
At about 4 characters a token, 1.5 trillion tokens.
- Count the crawls15.6 ÷ 1.5 = 10.4
About 10 months of the web, if you kept every page.
Answer1.5 trillion tokens a crawl, so about 10 crawls.
Nobody keeps every page. Most of a crawl is menus, spam and copies, and the next two parts throw them out. That's why FineWeb, a public dataset Hugging Face built from Common Crawl, used 96 crawls to reach 15 trillion tokens.
Corpus
The whole pile of text a model trains on, also called its training data or pretraining mix. It's measured in tokens. In English a token is about 4 characters: Llama 3's tokenizer averages 3.94.
tokens ≈ characters ÷ 4
That 4 belongs to one tokenizer and one language. On the same English sample, Llama 2's tokenizer averages 3.17 characters a token, so the same text costs it 24% more tokens. A token count always comes with a tokenizer.
Your turn
English Wikipedia holds about 5.3 billion words, and a token is roughly three-quarters of an English word. What share of a 15.6-trillion-token pile could all of it fill?
Part 2. Throwing out the junk
Check a page against simple quality rules, and say what a trained filter adds.
Open a crawled page and most of it isn't writing you'd want a model to copy. A text extractor keeps the main block of prose and drops the menus, banners and footers around it. A small model then checks the language. Last come simple rules, and the best known are from DeepMind's Gopher model.
Four of Gopher's rules keep a page only if it has 50 to 100,000 words, an average word length of 3 to 10 characters, a letter in at least 80% of its words, and at least two of eight very common words: the, be, to, of, and, that, have, with.
Example 2.1
Four rules, one page
After extraction, a news page has 57 words, and their letters and punctuation add up to 237 characters. Every word contains a letter, and the page uses "the", "and" and "of". Do the four rules keep it?
- Check the length57 ≥ 50
Long enough, and far under 100,000.
- Check the word length237 ÷ 57 = 4.2
Between 3 and 10 characters, like ordinary prose.
- Check for letters57 ÷ 57 = 100%
The rule asks for 80%. Price lists and runs of symbols fail here.
- Check the common words3 ≥ 2
Real sentences can't avoid words like these.
AnswerKeep it: it passes all four.
Each rule is cheap and catches one kind of junk. Rules only catch the obvious, so newer pipelines add a trained classifier. For FineWeb-Edu, Llama 3 70B scored 460,000 pages from 0 to 5 for how educational they were. A small, fast model learned from those scores, then rated all 15 trillion tokens of FineWeb.
Keeping the pages that scored 3 or more left 1.3 trillion tokens, about 9%. Models trained on that slice did better on knowledge and reasoning tests than models trained on the whole of FineWeb.
Quality filter
Any step that scores documents and drops the low ones: hand-written rules, or a trained classifier. Each one removes something you didn't want, and a little you did.
Your turn
A page reads "cheap shoes best price cheap shoes best price..." for 60 words. Which of the four rules throws it out?
Part 3. The same page, many times
Measure how alike two documents are by the runs of words they share.
The web repeats itself: cookie notices, licence text, one news story copied by fifty sites. A 2021 study of C4, a cleaned web dataset from Google, found one 61-word sentence repeated over 60,000 times. Exact copies are easy to catch: hash each document and keep one per hash. Near-copies need a softer test.
Example 3.1
Two versions of one story
A is "the council voted to close the old library on main street next spring". B is the same, with "summer" for "spring". Cut each into every run of 3 words in a row. What share of the runs do they have in common?
- Cut A into runs13 − 3 + 1 = 11
13 words give 11 runs, from "the council voted" to "street next spring".
- Cut B the same way11
Only its last run differs: "street next summer".
- Count shared and all runs10 shared, 12 in all
10 runs are in both. Together the two hold 12 different runs.
- Divide10 ÷ 12 = 0.83
Five-sixths of their runs match.
Answer0.83, so a pipeline would call B a near-copy of A.
A changed word only spoils the runs it sits in, at most 3 of them. In a long document that's a tiny share, so a page with a new date or a different ad still scores close to 1. Real pipelines use longer runs: FineWeb used 5 words, and aimed to catch pairs sharing 75% or more.
Jaccard similarity and MinHash
Shared runs divided by all the different runs is the Jaccard similarity. Comparing every pair among billions of documents is impossible, so MinHash squeezes each document's runs into a short fingerprint, and only documents whose fingerprints partly match get compared.
Jaccard = shared runs ÷ all runs
Your turn
Take sentence A again, but change "old" to "new" in the middle instead. What's the Jaccard similarity of the 3-word runs now?
Part 4. Choosing the mix
Turn a data mix into token budgets, and count how often a small source repeats.
After cleaning you have separate piles: web pages, code, maths, other languages. The last choice is how much of each the model reads. Meta published Llama 3's final mix: about 50% general knowledge, 25% maths and reasoning, 17% code and 8% other languages.
Example 4.1
A mix becomes budgets
A run will read 15 trillion tokens in Llama 3's mix. You also want a clean maths source of 50 billion tokens to fill 1% of the run. How big is each budget, and how often will the model see that source?
- General knowledge50% × 15T = 7.5T
Half the run.
- The other three3.75T, 2.55T, 1.2T
25%, 17% and 8% of 15 trillion. Maths gets far more than its share of the web, on purpose.
- The small source's budget1% × 15T = 150B
150 billion tokens to fill.
- Count the passes150B ÷ 50B = 3
The only way to fill it is to show the source 3 times.
AnswerBudgets of 7.5T, 3.75T, 2.55T and 1.2T tokens, and 3 passes over the maths source.
Giving a source more weight than its size means repeating it. A 2023 study of 400 training runs found that up to about 4 passes over the same data is nearly as good as fresh text. With more passes, each extra one is worth less, until it's worth nothing.
Data mix and epoch
The data mix is the share of training tokens each source gets. An epoch is one full pass over a source. Labs tune the mix by training small models on different mixes and comparing them.
passes = share × run tokens ÷ source tokens
Many runs change the mix near the end, and finish on a small set of the best maths, code and writing. When Meta tried this on Llama 3 8B, its scores on two maths tests, GSM8k and MATH, rose by 24.0% and 6.4%.
Your turn
You want a 5-billion-token source to make up 10% of a 1-trillion-token run. How many times will the model see each of its documents?
Where you'll meet it
A common question: how would you turn Common Crawl into training data? Walk the pipeline in order. Extract the main text, keep the languages you want, run cheap rules, then a trained quality filter. Remove exact and near copies. Strip personal details and test questions. Finish with the mix, and say how many times each source repeats.
The same tools guard test sets. Lee and others found that over 4% of the validation examples in standard datasets also sat in the training data. A model that has read the test can't be fairly scored on it, so teams check training data for overlapping word runs, as in part 3, before they trust a score.
STOP = {"the", "be", "to", "of", "and", "that", "have", "with"}
def keep(doc):
words = doc.split()
if not 50 <= len(words) <= 100_000:
return "drop: length"
if not 3 <= sum(len(w) for w in words) / len(words) <= 10:
return "drop: word length"
if sum(any(c.isalpha() for c in w) for w in words) < 0.8 * len(words):
return "drop: symbols"
if len(STOP & {w.lower() for w in words}) < 2:
return "drop: not prose"
return "keep"
def runs(text, n=3):
w = text.split()
return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
def jaccard(a, b):
return len(runs(a) & runs(b)) / len(runs(a) | runs(b))
print(keep("cheap shoes best price " * 15))
print(keep("---- 12.50 ---- 18.00 ---- 7.25 " * 10))
a = "the council voted to close the old library on main street next spring"
print(round(jaccard(a, a.replace("spring", "summer")), 2))
print(round(jaccard(a, a.replace("old", "new")), 2))
# drop: not prose
# drop: symbols
# 0.83
# 0.57Comparing every pair this way is fine for a few hundred documents. For billions, MinHash picks out the pairs worth comparing first.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
Why do training pipelines remove repeated documents, rather than let the model see good text more often?
2
Two 1,000-word pages are the same except for one word in the middle. Using 3-word runs, about how similar are they?
3
A collection of English books is 200 GB of plain text, one byte a character. You want it to be 10% of a 2-trillion-token run. About how many passes is that?
What to remember
1Counting text in tokens
Count text in tokens at about 4 English characters a token: 3 billion pages of 2,000 characters is about 1.5 trillion tokens.
2Throwing out the junk
Cheap rules drop the obvious junk, then a trained classifier keeps the best: FineWeb-Edu kept 1.3 of 15 trillion tokens, about 9%.
3The same page, many times
Two documents are near-copies when most of their word runs match: shared runs ÷ all runs, 10 ÷ 12 = 0.83 for two versions of one story.
4Choosing the mix
A source's passes are its budget ÷ its size: 1% of 15 trillion from a 50-billion-token source is 3 passes, and up to about 4 is nearly as good as fresh text.
Recap and going further
The lesson on one page
- Training text is measured in tokens
- English: about 4 characters a token
- Most of it comes from Common Crawl
- Keep the main text of each page
- Cheap rules, like Gopher's
- Then a trained classifier, as in FineWeb-Edu
- A hash catches exact copies
- Shared word runs catch near-copies
- MinHash makes that work across billions of pages
- Each source gets a share of the tokens
- Passes = share × run ÷ source size
- Up to about 4 passes is nearly as good as fresh text
- Personal details such as emails and phone numbers
- Questions from well-known tests, so scores stay fair
Try this
Score a slice of FineWeb yourself
Stream 500 documents from FineWeb's sample-10BT subset with the Hugging Face datasets library (streaming=True, so nothing big downloads). Score each one with HuggingFaceFW/fineweb-edu-classifier, the model behind FineWeb-Edu, then run this lesson's rules and 3-word Jaccard check over the same documents.
You need: Python with datasets, transformers and torch, or a free notebook such as Google Colab. A CPU is enough for 500 documents.
Look for
- What share of documents score 3 or more, the cut FineWeb-Edu used.
- How a page that scores 1 reads next to one that scores 4.
- How few documents the simple rules drop, since FineWeb already ran rules like them.
- How many characters each token covers, against the rule of thumb of 4.
The words, in plain language
- Corpus
- The whole pile of text a model trains on, also called its training data or pretraining mix. It's measured in tokens. In English a token is about 4 characters: Llama 3's tokenizer averages 3.94.
- Quality filter
- Any step that scores documents and drops the low ones: hand-written rules, or a trained classifier. Each one removes something you didn't want, and a little you did.
- Jaccard similarity and MinHash
- Shared runs divided by all the different runs is the Jaccard similarity. Comparing every pair among billions of documents is impossible, so MinHash squeezes each document's runs into a short fingerprint, and only documents whose fingerprints partly match get compared.
- Data mix and epoch
- The data mix is the share of training tokens each source gets. An epoch is one full pass over a source. Labs tune the mix by training small models on different mixes and comparing them.
Read next
- FineWeb: decanting the web for the finest text data at scaleHugging FaceEach filtering and deduplication step behind a 15-trillion-token dataset, with small training runs showing what each one bought.
- Deduplicating Training Data Makes Language Models BetterKatherine Lee and others, 2021The study behind the 61-word sentence repeated over 60,000 times, and the ten-fold drop in copied text.
- The Llama 3 Herd of ModelsMeta, 2024Section 3.1 covers the web cleaning, deduplication, data mix and end-of-run data behind Llama 3.
Training and Tuning overviewSpotted a mistake? Tell us.