Choose where to start.
Two subjects, and what's inside each.
-
Start with lesson 1
AI engineering
Learn how models work, then build on them.
-
Start with lesson 1
Quant engineering
Learn the maths and data behind quant roles.
See a lesson from the inside.
Why chat models miscount the r's in strawberry, one step at a time.
32Tokens and Embeddings
Letters hide inside tokens
Ask some chat models how many r’s are in “strawberry”, and they answer two. Look at what the model actually receives, and the slip makes sense. In this lesson, ␣ marks a space.
Example 1.1
Counting the r’s in strawberry
GPT-2’s tokenizer reads “How many r’s are in strawberry?”. What does the model get for the word, and where are the r’s?
- 1Cut into tokensHow␣many␣r's␣are␣in␣strawberry?8 tokens. Most words carry the space in front of them.
- 2Find the word␣strawberry → 41236The whole word, space included, is one token in GPT-2’s vocabulary.
- 3Look for the letters41236One number. The three r’s are inside it, out of sight.
AnswerOne token, 41236. To count its r’s, the model must have learned how that token is spelled.
Tokenizer quirks: numbers, spelling, languages33
A model can learn spellings, but only second-hand, from text where words get spelled out, rhymed or misspelled. That knowledge is patchy across tens of thousands of tokens, so letter tasks go wrong where meaning tasks go fine. Spelling the word out first helps, because then each letter becomes its own token.
Token boundary
Where one token ends and the next begins.
␣strawberry → 41236: one ID, 10 letters
Your turn
GPT-2 cuts “␣bookkeeper” into “␣book” (ID 1492) and “keeper” (ID 13884). Asked how many e’s the word has, what does the model have to go on?
- (a)Two IDs, plus whatever it learned elsewhere about how each is spelled
- (b)Ten letters, so it can count the three e’s directly
- (c)One byte token per letter
- (d)Nothing, so it always answers zero
32Tokens and Embeddings
Letters hide inside tokens
Ask some chat models how many r’s are in “strawberry”, and they answer two. Look at what the model actually receives, and the slip makes sense. In this lesson, ␣ marks a space.
Example 1.1
Counting the r’s in strawberry
GPT-2’s tokenizer reads “How many r’s are in strawberry?”. What does the model get for the word, and where are the r’s?
- 1Cut into tokensHow␣many␣r's␣are␣in␣strawberry?8 tokens. Most words carry the space in front of them.
- 2Find the word␣strawberry → 41236The whole word, space included, is one token in GPT-2’s vocabulary.
- 3Look for the letters41236One number. The three r’s are inside it, out of sight.
AnswerOne token, 41236. To count its r’s, the model must have learned how that token is spelled.
More than lessons.
Put what you read to work.
Used by 25,000+ engineers from
Get your pass.
Frequently asked questions
What is insideout?
Lessons that show how AI and quant systems work, with every step of every example written out. Around them: daily practice, interview prep for six roles, and projects you build yourself.
Who is it for?
Engineers who want to understand the tools they use: developers moving into AI engineering, and anyone heading into quant research, trading or development.
Where do I start?
On Learn, pick AI or quant engineering and open the first lesson. Each module starts from the basics, and Tracks show the order we'd take them in.
What's free?
The first three lessons of every module, with no account. A free account adds Practice, Projects and Interview, and saves your progress on any device.
What does Pro add?
Every lesson in every module, every interview round, and unlimited practice: more sets, any topic on its own, and timed runs.
How much maths do I need?
School algebra is enough to start. Each idea begins in plain words with small numbers, and every step is shown. Some Python helps with the code.
Does it work on my phone?
Yes. Lessons, Practice and Interview all work on a phone. Projects need a computer.
What does Pro cost?
$5.99 a month, billed yearly. You can also pay monthly, every three months or once for life, and cancel any time.
Something else? Write to us at help@insideout.sh