Your own BPE tokenizer and a bigram model on its tokens, scored in bits per byte so every vocabulary size, and gzip, compare fairly.
Download two public-domain novels by one author from Project Gutenberg as plain text (Pride and Prejudice and Emma work), then train on one and hold out the other.
Write byte-level BPE yourself: split the text into words, count them, and merge the most common pair until you have 256 + 1,000 tokens.
Check that encoding then decoding gives back every held-out line exactly, and compare bytes per token on the held-out book with GPT-2's tokenizer in tiktoken.
Build a bigram model over your tokens by counting, choosing the smoothing on a slice of the training book.
Train the same bigram in PyTorch as a 64-wide embedding table plus an output layer, and list the nearest neighbours of five tokens by cosine.
Turn each held-out loss into bits per byte at vocabularies of 256, 512 and 1,256, and put gzip's figure beside them.
How you'll know it worksA table of bytes per token and held-out bits per byte for each vocabulary, counted and trained, with gzip as a row, and a round trip that's exact on every held-out line.
Python, tiktoken and PyTorch on a laptop CPU. The books are free plain-text downloads from Project Gutenberg. Uses lessons 01 to 11.
Project: A tokenizer and a bigram model on one book
From insideout, Tokens and Embeddings: Your own BPE tokenizer and a bigram model on its tokens, scored in bits per byte so every vocabulary size, and gzip, compare fairly.
Steps:
1. Download two public-domain novels by one author from Project Gutenberg as plain text (Pride and Prejudice and Emma work), then train on one and hold out the other.
2. Write byte-level BPE yourself: split the text into words, count them, and merge the most common pair until you have 256 + 1,000 tokens.
3. Check that encoding then decoding gives back every held-out line exactly, and compare bytes per token on the held-out book with GPT-2's tokenizer in tiktoken.
4. Build a bigram model over your tokens by counting, choosing the smoothing on a slice of the training book.
5. Train the same bigram in PyTorch as a 64-wide embedding table plus an output layer, and list the nearest neighbours of five tokens by cosine.
6. Turn each held-out loss into bits per byte at vocabularies of 256, 512 and 1,256, and put gzip's figure beside them.
How you'll know it works: A table of bytes per token and held-out bits per byte for each vocabulary, counted and trained, with gzip as a row, and a round trip that's exact on every held-out line.
Runs on: Python, tiktoken and PyTorch on a laptop CPU. The books are free plain-text downloads from Project Gutenberg.
Help me build this on my machine, one step at a time. Ask what I've tried first, give hints before full answers, and check my work against "How you'll know it works".