insideout Get Pro

11 · Turning logits into choices

Pro

Transformers, lesson 11

Turning logits into choices

Greedy picks, and how temperature sharpens or flattens the odds.

You need "Autoregressive generation": run the model, pick a token, append it, repeat. This lesson is about the pick.

Part 1. Always take the top

After "The cat sat on the", the model's top candidates are mat 0.41, floor 0.20, couch 0.12, bed 0.08. The simplest rule is to take the most likely token every time. Here that's "mat", no dice involved.

Greedy decoding

Taking the top-ranked token at each step, with no dice roll. In code it's an argmax: max over the probabilities, or over the logits, since softmax keeps their order.

Your turn

You run greedy decoding on the same prompt, with the same model, five times. What do you get?

Greedy uses no randomness at all: the same input gives the same probabilities, and the top one is the top one. The randomness people associate with chatbots comes from sampling, not from the model. (On shared servers, tiny rounding differences can still flip a near-tie. "Seeds, reproducibility and picking settings" covers that.)

That makes greedy a good fit for short answers where there's one right output: pulling a date out of an email, choosing a label from a fixed list, filling in a function name. Many APIs treat temperature 0 as greedy, and the second half of this lesson shows why.

Part 2. Why greedy repeats itself

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Why greedy repeats itself
  2. The best step isn't the best sentence
  3. Divide the scores first
  4. Sharper and flatter
See Pro plansI have Pro: sign in
Next lessonCutting the tail with top-k and top-pSample only from the likeliest tokens to avoid rare nonsense. Start lesson 12

Transformers overviewSpotted a mistake? Tell us.