Training and Tuning, lesson 7
Rewarding checkable answers
Reinforcement learning on maths and code, where a program grades the answer.
You know RLHF from "Learning from preferences": a model writes answers, something scores them, and training nudges it toward higher scores.
Part 1. Answers you can check
A reward model is a guess about what people would like, and guesses can be fooled. Some tasks don't need a guess. "What is 17 × 24?" has one right answer, 408. A function that should sort a list either passes its tests or it doesn't.
So score with a checker instead: 1 if the final answer matches, 0 if it doesn't. For code, run the tests: 1 if they pass. No person in the loop, and no reward model to flatter.
Your turn
Which task fits a checkable reward best?
Verifiable reward
A reward computed by a program that checks the answer: a string match, a unit test, a proof checker. Reinforcement learning on these is often called RLVR, reinforcement learning with verifiable rewards.
Part 2. Try several, learn from the better ones
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Try several, learn from the better ones
- Why models now think longer
- Reward hacking
Training and Tuning overviewSpotted a mistake? Tell us.