Tokens and Embeddings, lesson 1
Text as bytes, bytes as tokens
How text becomes a list of integer IDs a model can read.
A language model reads a list of whole numbers, never letters. This lesson follows a sentence down to those numbers: as words, then as bytes, then as tokens. You need to know that a network takes numbers in ("One neuron"), and a little Python.
By the end, you can:
Part 1. Text becomes a list of IDs
Turn a sentence into token IDs with a vocabulary, and spot the unknown word.
A network only takes numbers. So before a model can read a sentence, something has to cut the text into pieces and swap each piece for a number. The simplest cut is at the spaces, with one number for each word on a fixed list.
Example 1.1
Eight entries, one sentence
A tokenizer knows eight entries, numbered from 0: the, cat, sat, on, mat, dog, a, and a last one, unknown, for anything else. Which numbers does it give "the dog sat on the rug"?
- Cut at the spacesthe | dog | sat | on | the | rug
Six pieces, one per word.
- Look up each piecethe → 0, dog → 5, sat → 2, on → 3, the → 0
A word's number is its place on the list, counting from 0. Both copies of "the" get the same 0.
- Handle the missing piecerug → 7
"rug" isn't on the list, so it gets the unknown entry, number 7.
Answer0, 5, 2, 3, 0, 7.
The model never sees "the". It sees 0, every time. Whatever it learns about 0 in one sentence carries over to every other sentence with "the" in it. That's why the list is fixed: if "the" were 0 today and 4 tomorrow, nothing learned would carry over.
Token and vocabulary
A token is one of the pieces a tokenizer cuts text into. The vocabulary is the fixed list of tokens it knows. A token's place on that list is its token ID, the number the model receives.
the dog sat → 0 5 2
The unknown entry is the weak spot. A typo, a new product name and a word in Welsh would all become 7, and the model couldn't tell them apart. A longer list only helps so much: English alone has hundreds of thousands of words, before names, code and every other language.
Cutting into single characters fixes most of that, since 26 letters and a space cover this whole sentence. But the list of IDs gets long: "the dog sat on the rug" is 6 words and 22 characters, so 22 IDs instead of 6.
Your turn
With the same list (the 0, cat 1, sat 2, on 3, mat 4, dog 5, a 6, unknown 7), which IDs does "a cat sat on a hat" get?
Part 2. Every character is stored as bytes
Write a word as UTF-8 bytes and count them.
Characters aren't the bottom layer either. Computers store text as bytes: whole numbers from 0 to 255. The rule nearly everyone uses to turn characters into bytes is called UTF-8. Plain English letters, digits and punctuation take one byte each: "A" is 65, "a" is 97 and a space is 32.
Example 2.1
The bytes of café
How many bytes does UTF-8 use for "café", and what are they?
- Write the plain lettersc → 99, a → 97, f → 102
Plain English letters take one byte each.
- Write the accented letteré → 195 169
é isn't plain English, so UTF-8 writes it as two bytes.
- Count3 + 2 = 5
4 characters, 5 bytes.
Answer5 bytes: 99, 97, 102, 195, 169.
A byte has room for 256 values, and UTF-8 keeps 0 to 127 for plain English letters, digits, punctuation and a few control codes. Every other character becomes a run of two to four bytes, each 128 or more, so a run can't be mixed up with plain letters. The first byte of a run says how long it is.
Byte and UTF-8
A byte is a whole number from 0 to 255. UTF-8 is the standard rule for writing each character as one to four bytes, and for reading them back. Nearly every web page is stored this way.
café → 99 97 102 195 169
So a character count and a byte count only match for plain English. In Python, len("café") is 4, but len("café".encode("utf-8")) is 5. And half a character means nothing: the byte 195 on its own isn't any letter, and decoding it alone raises an error.
Your turn
How many bytes does UTF-8 use for "über"?
Part 3. 256 tokens that cover everything
Encode any text with a byte-level tokenizer, and say what it costs.
Now put the two ideas together. Words meet unknowns. Characters need a huge list: Unicode, the standard catalogue of characters, holds over 150,000. Bytes have only 256 values. So give the vocabulary exactly 256 tokens, one per byte, with token n standing for byte n.
Example 3.1
An emoji nobody planned for
A byte-level tokenizer has only its 256 starting tokens. Which tokens does it give "hi 🙂"?
- Write the text as bytesh 104, i 105, space 32, 🙂 240 159 153 130
The emoji takes four bytes in UTF-8.
- Swap each byte for its token104 105 32 240 159 153 130
Token n stands for byte n, so the IDs are the bytes themselves.
- Count3 + 4 = 7
One token for each plain character, four for the emoji.
Answer7 tokens, and none of them unknown.
Every piece of text is bytes, and every byte has a token, so nothing can be unknown: not a new language, a typo, a line of code, or a character added to Unicode next year. A word list can't promise that, and nor can a list of the characters seen in training.
Byte-level tokenizer
A tokenizer whose starting vocabulary is the 256 byte values, so it can encode any text and never needs an unknown token. The tokenizers of GPT-2, GPT-4 and GPT-4o all start this way.
256 tokens: one per byte, 0 to 255
The catch is length. "Hello, world!" is 13 bytes, so 13 byte tokens. GPT-2's tokenizer gives it 4: "Hello" is 15496, the comma 11, " world" with its space 995, and "!" 0. Models are priced and limited by the token, so long lists cost money and room.
GPT-2 numbers its 256 byte tokens in its own order, which is why "!" is 0 and not 33. Nearly all the rest of its 50,257 tokens are longer runs of bytes. The next lesson shows how those runs are chosen.
Your turn
A byte-level tokenizer with only its 256 starting tokens meets "Καλημέρα", Greek for "good morning". It was never shown any Greek. What comes out?
Where you'll meet it
If you're asked why models don't give every word its own token, answer with the trade-off. Words leave unknowns and need a huge list. Characters and bytes never run out but make long sequences. Then name the fix: start from the 256 byte tokens and add tokens for common runs, so nothing is unknown and lists stay short.
Model APIs bill by the token and cap each request at a number of tokens, so engineers count tokens, not characters. Text from users, files and web pages arrives as UTF-8 bytes. When the bytes are read back with the wrong rule, you get garbled words like "café": the two bytes of é, read as two separate characters.
for text in ["cat", "café", "über", "€5", "hi 🙂"]:
data = text.encode("utf-8")
print(text, len(text), len(data), list(data)) # characters, bytes, the bytes
print(bytes([99, 97, 102, 195, 169]).decode("utf-8"))
print(bytes([99, 97, 102, 195, 169]).decode("latin-1")) # the wrong rule
# cat 3 3 [99, 97, 116]
# café 4 5 [99, 97, 102, 195, 169]
# über 4 5 [195, 188, 98, 101, 114]
# €5 2 4 [226, 130, 172, 53]
# hi 🙂 4 7 [104, 105, 32, 240, 159, 153, 130]
# café
# cafélen counts characters on a string and bytes on its encoding. Reading UTF-8 with the wrong rule doesn't raise an error, it prints the wrong letters, so check the encoding whenever text looks garbled.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A word tokenizer has a vocabulary of the 50,000 most common English words. Someone types "unfriendable". What does the model receive?
2
How many tokens does a byte-level tokenizer, with only its 256 starting tokens, give "€20"?
3
You take the first 4 bytes of "café", which are 99, 97, 102 and 195, and decode them as UTF-8. What happens?
What to remember
1Text becomes a list of IDs
A tokenizer cuts text into pieces and swaps each for its place on a fixed vocabulary: "the dog sat on the rug" became 0 5 2 3 0 7, and every word not on the list gets the same unknown ID.
2Every character is stored as bytes
UTF-8 writes each character as one to four bytes, each a number from 0 to 255: "café" is 4 characters and 5 bytes.
3256 tokens that cover everything
A byte-level tokenizer starts with 256 tokens, one per byte, so nothing is ever unknown, but "Hello, world!" takes 13 byte tokens against GPT-2's 4.
Recap and going further
The lesson on one page
- A fixed list of pieces, each with an ID
- Every unlisted word gets the one unknown ID
- Words: short lists, many unknowns
- Characters: fewer unknowns, long lists
- Whole numbers from 0 to 255
- UTF-8 writes each character as one to four
- Plain English: one
- Accents and Greek: two
- Most Chinese, Japanese and Hindi: three
- Most emoji: four
- Starts with 256 tokens, one per byte
- Never meets an unknown
- Lists get long, so common runs get glued
Try this
Look at the bytes, then the tokens
In Python, print list("café".encode("utf-8")), then do the same for a word in Greek, a word in Hindi and an emoji. Try bytes([195]).decode("utf-8") to see what half a character does. Then load GPT-2's tokenizer with tiktoken.get_encoding("gpt2"), encode one sentence in each language, and compare its byte count with its token count.
You need: Python 3 and pip install tiktoken (it downloads the encoding file once).
Look for
- How many bytes each script takes per character.
- The error you get from decoding half of a two-byte character.
- Whether "Hello, world!" really comes out as 15496, 11, 995, 0.
- How the ratio of bytes to tokens changes from English to the other languages.
The words, in plain language
- Token and vocabulary
- A token is one of the pieces a tokenizer cuts text into. The vocabulary is the fixed list of tokens it knows. A token's place on that list is its token ID, the number the model receives.
- Byte and UTF-8
- A byte is a whole number from 0 to 255. UTF-8 is the standard rule for writing each character as one to four bytes, and for reading them back. Nearly every web page is stored this way.
- Byte-level tokenizer
- A tokenizer whose starting vocabulary is the 256 byte values, so it can encode any text and never needs an unknown token. The tokenizers of GPT-2, GPT-4 and GPT-4o all start this way.
Read next
- The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)Joel SpolskyCharacters, code points and encodings, explained for programmers.
- Unicode HOWTOPython documentationHow Python stores text, and how encode and decode turn it into bytes and back.
- UTF-8WikipediaThe byte patterns for one to four bytes, in one table, and how a first byte says how long its run is.
Tokens and Embeddings overviewSpotted a mistake? Tell us.