A character-level GPT you write line by line in PyTorch, trained on your laptop and sampled with your own generation loop.
Download Tiny Shakespeare (about 1.1 million characters, in Karpathy's char-rnn repository), give each character an ID, split it 90/10, and train a bigram baseline.
Write one causal attention head yourself, with queries, keys, values, the square-root scaling and the mask, and check it against PyTorch's scaled_dot_product_attention.
Build the block (layer norm, multi-head attention, MLP and residual connections), stack four with learned position embeddings, and train with AdamW.
Train again with the position embeddings taken out, and compare the two validation losses.
Write the generation loop with temperature, top-k and top-p, and sample 500 characters at three settings from a fixed seed.
How you'll know it worksValidation loss under 2.0 (the bigram sits near 2.5), your attention head within 1e-5 of PyTorch's, and the same seed giving the same 500 characters twice.
Python and PyTorch, on a laptop CPU or Apple's mps device. A model of about 800,000 parameters trains in under half an hour. Uses lessons 02 to 17.
Project: A small GPT on Shakespeare
From insideout, Transformers: A character-level GPT you write line by line in PyTorch, trained on your laptop and sampled with your own generation loop.
Steps:
1. Download Tiny Shakespeare (about 1.1 million characters, in Karpathy's char-rnn repository), give each character an ID, split it 90/10, and train a bigram baseline.
2. Write one causal attention head yourself, with queries, keys, values, the square-root scaling and the mask, and check it against PyTorch's scaled_dot_product_attention.
3. Build the block (layer norm, multi-head attention, MLP and residual connections), stack four with learned position embeddings, and train with AdamW.
4. Train again with the position embeddings taken out, and compare the two validation losses.
5. Write the generation loop with temperature, top-k and top-p, and sample 500 characters at three settings from a fixed seed.
How you'll know it works: Validation loss under 2.0 (the bigram sits near 2.5), your attention head within 1e-5 of PyTorch's, and the same seed giving the same 500 characters twice.
Runs on: Python and PyTorch, on a laptop CPU or Apple's mps device. A model of about 800,000 parameters trains in under half an hour.
Help me build this on my machine, one step at a time. Ask what I've tried first, give hints before full answers, and check my work against "How you'll know it works".