insideout Get Pro

06 · Learning from preferences

Pro

Training and Tuning, lesson 6

Learning from preferences

RLHF and DPO, trained on people's choices between two answers.

You know instruction tuning from "Instruction tuning": train on one good answer per request, scoring only the answer tokens.

Part 1. Comparing is easier than writing

Ask someone to write the perfect reply to "Explain why the sky is blue to a ten-year-old" and they'll be at it a while. Show them two replies and ask which is better, and they can usually tell you straight away.

Example 1.1

One preference

  1. promptExplain why the sky is blue to a ten-year-old.

    One request.

  2. ASunlight has every colour in it. Air bounces blue light around the most, so blue reaches you from every part of the sky.
    BRayleigh scattering is inversely proportional to the fourth power of wavelength.

    The model writes two answers (or more).

  3. chosen: Arejected: B

    A person picks one. B is correct, but it isn't for a ten-year-old. That judgement is hard to write down as a rule and easy to make by eye.

Your turn

Instruction tuning already trains on good answers. What can comparisons teach that it can't?

Instruction tuning sees one answer per request and pushes toward it. A comparison has a direction: more like this, less like that. It can push a model away from habits (padding, preaching, confident errors) that no single good example points at.

OpenAI's InstructGPT (2022) did this at scale, with people ranking several answers per prompt. People preferred the answers of the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3, a model more than 100 times bigger.

Preference pair

One prompt with two answers, marked chosen and rejected. Preference data is a big pile of these, labelled by people or by another model acting as judge.

Part 2. A model that scores answers

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A model that scores answers
  2. RLHF: chase the score, on a leash
  3. DPO: skip the reward model
  4. Your own rules as preference pairs
See Pro plansI have Pro: sign in
Next lessonRewarding checkable answersReinforcement learning on maths and code, where a program grades the answer. Start lesson 7

Training and Tuning overviewSpotted a mistake? Tell us.