Tokens and Embeddings, lesson 3
Tokenizer quirks: numbers, spelling, languages
Why models miscount letters, split numbers oddly and charge more for some languages.
A model reads token IDs, so whatever sits inside a token is hard for it to see. This lesson looks at where that shows: in spelling, numbers, spaces and languages. You need BPE from "Training a BPE tokenizer". The examples use GPT-2's and GPT-4o's real tokenizers, and the Tiktokenizer web app shows the same cuts.
By the end, you can:
- Find which tokens a word becomes, and say why counting its letters is hard.
- Predict how a tokenizer cuts a number, with merges or with fixed chunks.
- Tell why the same word can get different IDs, and why a prompt shouldn't end in a space.
- Count bytes and tokens for the same text in different languages, and price the gap.
Part 1. Letters hide inside tokens
Find which tokens a word becomes, and say why counting its letters is hard.
Ask some chat models how many r's are in "strawberry", and they answer two. Look at what the model actually receives, and the slip makes sense. In this lesson, ␣ marks a space.
Example 1.1
Counting the r's in strawberry
GPT-2's tokenizer reads "How many r's are in strawberry?". What does the model get for the word, and where are the r's?
- Cut into tokensHow | ␣many | ␣r | 's | ␣are | ␣in | ␣strawberry | ?
8 tokens. Most words carry the space in front of them.
- Find the word␣strawberry → 41236
The whole word, space included, is one token in GPT-2's vocabulary.
- Look for the letters41236
One number. The three r's are inside it, out of sight.
AnswerOne token, 41236. To count its r's, the model must have learned how that token is spelled.
A model can learn spellings, but only second-hand, from text where words get spelled out, rhymed or misspelled. That knowledge is patchy across tens of thousands of tokens, so letter tasks go wrong where meaning tasks go fine. Spelling the word out first helps, because then each letter becomes its own token.
Token boundary
Where one token ends and the next begins. A model gets each token as one ID, so it can see the boundaries between tokens but not the letters inside one. Tasks about letters work better when each letter is its own token.
␣strawberry → 41236: one ID, 10 letters
Where the cuts fall depends on what's around the word. At the very start of a text there's no space, and GPT-2 cuts "strawberry" into st, raw and berry. Different cuts give different IDs for the same word, and the model has to learn that they mean the same thing.
Your turn
GPT-2 cuts "␣bookkeeper" into "␣book" (ID 1492) and "keeper" (ID 13884). Asked how many e's the word has, what does the model have to go on?
Part 2. Numbers come out in odd chunks
Predict how a tokenizer cuts a number, with merges or with fixed chunks.
Merges come from whatever is common in the training text, and that includes numbers. A year like 2020 turns up so often that GPT-2 learned it as one token. Rarer numbers get cut wherever the merges happen to fall.
Example 2.1
Four numbers in GPT-2
How does GPT-2's tokenizer cut 2020, 2023, 1234567 and 1000000, each with a space in front, as in running text?
- Cut 2020␣2020
Common enough to be one token.
- Cut 2023␣20 | 23
A year only three later splits in two.
- Cut 1234567␣123 | 45 | 67
Three digits, then two, then two.
- Cut 1000000␣100 | 0000
Three digits, then four.
Answer1, 2, 3 and 2 tokens, in four different shapes.
Now picture adding numbers. In "␣123 | 45 | 67" the millions digit shares a token with two others, while in "␣2020" a whole year is one ID. The model has to learn place value separately for every way the merges might cut a number, which makes sums harder to learn.
Digit splitting
A fixed rule for cutting numbers before any merges apply. Llama 1 and 2 split every number into single digits. The tokenizers of GPT-4 and GPT-4o cut runs of digits into chunks of up to three, starting from the left.
1234567 → 123 | 456 | 7
Chunks from the left don't line up with place value. 1234567 becomes 123 456 7, while people group it from the right: 1,234,567. One study found that adding the commas, which makes the tokenizer cut from the right, made GPT-3.5 and GPT-4 much better at sums.
Your turn
GPT-4o's tokenizer cuts runs of digits into chunks of three from the left. How does it cut 12345678?
Part 3. A space is part of the word
Tell why the same word can get different IDs, and why a prompt shouldn't end in a space.
Before any merges, GPT-2's tokenizer cuts text into chunks: a word with the space in front of it, a run of digits, or a run of punctuation. Merges never cross from one chunk into the next. In running text nearly every word follows a space, so the spaced version is the one the merges learn best.
Example 3.1
Five IDs for cat
Which GPT-2 tokens do "cat", "␣cat", "Cat", "␣Cat" and "␣CAT" get?
- Lower case, with and without a space␣cat → 3797, cat → 9246
A space is a byte like any other, so these are different byte strings.
- A capital C␣Cat → 5181, Cat → 21979
C is byte 67 and c is byte 99, so a capital makes new strings too.
- All capitals␣CAT → 38348
Common enough to get a token of its own as well.
AnswerFive different IDs, each learned on its own.
To the model these are five unrelated numbers, and it has to learn from use that they mean the same animal. The common forms get plenty of practice. Rarer ones, like a word in capitals or at the very start of a text, get less, so the model handles them less well.
Pre-tokenization
Cutting text into chunks before BPE runs: usually a word with the space in front of it, a run of digits, or a run of punctuation. Merges only happen inside a chunk, so no token runs across two words.
The cat sat → The | ␣cat | ␣sat
That leading space sets a trap. End a prompt with "The capital of France is ", space included, and its last token is a lone space, ID 220. In training text that space nearly always came inside the next token, "␣Paris" (6342). Now the model has to go on with "Paris" minus its space, a token it rarely saw there.
Your turn
GPT-2 tokenizes "Cat cat cat". How many tokens come out, and how many different IDs?
Part 4. Some languages cost more
Count bytes and tokens for the same text in different languages, and price the gap.
Merges are learned from training text, and GPT-2's was almost all English: its makers removed pages in other languages. So English words merged into whole tokens, while other scripts got few merges and stay close to raw bytes. Many of them also start with two or three bytes per character.
Example 4.1
Good morning, in GPT-2's tokens
How many bytes and how many GPT-2 tokens does "good morning" take in English, Spanish, Greek and Hindi?
- English: Good morning12 bytes → 2 tokens
One byte per character, and both words are in the vocabulary.
- Spanish: Buenos días12 bytes → 5 tokens
Only í takes two bytes, but the words were rare in training, so they split: Bu, enos, ␣d, í, as.
- Greek: Καλημέρα16 bytes → 11 tokens
Two bytes per letter, and few merges: 11 tokens for 8 letters.
- Hindi: सुप्रभात24 bytes → 15 tokens
Three bytes per character, and fewer merges still.
AnswerThe same greeting: 2 tokens in English, 15 in Hindi.
Fertility
The average number of tokens a tokenizer uses per word. The lower it is for a language, the cheaper, faster and roomier a model is in that language. Measure it on real text, with the tokenizer of the model you'll use.
tokens ÷ words: Καλημέρα, 11 ÷ 1 = 11
Everything priced or limited per token gets worse for the costlier language. The bill is higher, less of a long document fits in the model's input, and replies take longer, because the reply comes out one token after another. Bigger vocabularies shrink the gap: GPT-4o's needs 5 tokens for the Hindi, not 15.
Your turn
With GPT-4o's tokenizer, "Good morning" is 2 tokens and "Καλημέρα" is 4. A bot sends the same number of greetings in each language, priced per token. How does the Greek cost compare?
Where you'll meet it
"Why can't a model count the r's in strawberry?" tests whether you know what a model reads. Answer with tokens: the word arrives as one ID or a few, and the letters are inside them. Then give the fixes: have the model spell the word out first, or hand the counting to code.
Before you estimate a budget, count tokens with the tokenizer of the model you'll actually use, on real text in each language you serve. OpenAI's tiktoken does it in a few lines, and other providers offer token counters too. Strip trailing spaces from prompt templates, and write numbers in one consistent format.
import tiktoken # pip install tiktoken
gpt2 = tiktoken.get_encoding("gpt2")
gpt4o = tiktoken.get_encoding("o200k_base")
def pieces(enc, text):
return [enc.decode([t]) for t in enc.encode(text)]
print(pieces(gpt2, " 1234567"), pieces(gpt4o, " 1234567"))
for text in ["Good morning", "Καλημέρα", "सुप्रभात"]:
print(text, len(text.encode("utf-8")), len(gpt2.encode(text)), len(gpt4o.encode(text)))
# [' 123', '45', '67'] [' ', '123', '456', '7']
# Good morning 12 2 2
# Καλημέρα 16 11 4
# सुप्रभात 24 15 5Each line prints the text, its bytes, then its tokens in GPT-2 and in GPT-4o. tiktoken downloads each encoding the first time you load it.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
Which request is hardest for a model that reads tokens?
2
You end a prompt with "The capital of France is ", including a space at the end. What's the likely effect?
3
A support bot answers the same questions in English and in Hindi. Its tokenizer's merges came mostly from English text. What happens on the Hindi side?
4
Which way of splitting numbers makes adding them easiest to learn?
What to remember
1Letters hide inside tokens
A model sees one ID per token, not letters: "␣strawberry" is the single GPT-2 token 41236, so counting its r's depends on what the model learned about that token.
2Numbers come out in odd chunks
Numbers split wherever the merges fell, like ␣123 | 45 | 67 in GPT-2; GPT-4o cuts digits into threes from the left, and Llama 1 and 2 into single digits.
3A space is part of the word
A word carries the space in front of it, so "cat", "␣cat" and "␣Cat" are three different tokens, and a prompt shouldn't end with a space.
4Some languages cost more
The same text can take far more tokens in one language than another: "good morning" is 2 GPT-2 tokens in English and 15 in Hindi, so it costs more and fits less.
Recap and going further
The lesson on one page
- Letters hide inside tokens
- ␣strawberry is one GPT-2 token
- Spelling a word out first helps
- Merges cut numbers into odd chunks
- Llama 1 and 2: one token per digit
- GPT-4 and GPT-4o: chunks of up to three
- A word carries the space before it
- cat, ␣cat and ␣Cat are different tokens
- End prompts without a space
- More bytes per character, fewer merges
- More tokens: a bigger bill, less fits, slower replies
- Bigger vocabularies narrow the gap
Try this
Find the quirks in real tokenizers
With tiktoken, load GPT-2's encoding (gpt2) and GPT-4o's (o200k_base). In each, encode "strawberry", " strawberry" and " Strawberry", the numbers 2020, 2024, 1999 and 1234567, and "good morning" in English, Spanish, Greek, Japanese and Hindi. Print each token's bytes with enc.decode_single_token_bytes.
You need: Python and pip install tiktoken. For a no-install version, the Tiktokenizer web app shows tokens as you type.
Look for
- How many pieces each strawberry becomes, and where the cuts fall.
- How each encoding cuts a long number, and where GPT-4o puts the space.
- Tokens per greeting in each language, and how much the bigger vocabulary narrows the gap.
- Whether a trailing space ends up as a token of its own.
The words, in plain language
- Token boundary
- Where one token ends and the next begins. A model gets each token as one ID, so it can see the boundaries between tokens but not the letters inside one. Tasks about letters work better when each letter is its own token.
- Digit splitting
- A fixed rule for cutting numbers before any merges apply. Llama 1 and 2 split every number into single digits. The tokenizers of GPT-4 and GPT-4o cut runs of digits into chunks of up to three, starting from the left.
- Pre-tokenization
- Cutting text into chunks before BPE runs: usually a word with the space in front of it, a run of digits, or a run of punctuation. Merges only happen inside a chunk, so no token runs across two words.
- Fertility
- The average number of tokens a tokenizer uses per word. The lower it is for a language, the cheaper, faster and roomier a model is in that language. Measure it on real text, with the tokenizer of the model you'll use.
Read next
- Language Model Tokenizers Introduce Unfairness Between LanguagesPetrov and colleagues, 2023Measures how many more tokens the same text takes in different languages, and what that costs.
- Tokenization counts: the impact of tokenization on arithmetic in frontier LLMsSingh and Strouse, 2024Cutting numbers into threes from the right, by adding commas, made GPT-3.5 and GPT-4 much better at sums than the default cut from the left.
- TiktokenizerA free web appType any text and see each model's tokens and IDs as you go.
Tokens and Embeddings overviewSpotted a mistake? Tell us.