insideout Get Pro

02 · Rubrics

Free

Evaluation, lesson 2

Rubrics

Break quality into parts a person or a model can score the same way twice.

A single score from 1 to 5 can't tell you what to fix, and two graders rarely give the same one. A rubric splits quality into parts and says how to mark each, so that a person or a model marks the same way twice. You need criteria and pass rates from "Specifying a good output".

By the end, you can:

  1. Split one quality score into parts that each point at a fix.
  2. Count how often two graders agree, and raise it by describing each level.
  3. Turn a scale into yes or no questions, and weigh what that gains and costs.
  4. Combine parts into one score with weights, and gate the parts you can't ship without.

Part 1. One score hides the reason

Split one quality score into parts that each point at a fix.

Two replies to "How do I change my delivery address?" both get 3 out of 5. Reply A is correct but runs to four paragraphs. Reply B is short and friendly, and names a menu that doesn't exist. The same 3 needs two different fixes, and it can't say which.

Example 1.1

Four parts, two replies

Mark both replies yes or no on four parts. Correct: the steps work. Complete: it says what happens next. Clear: easy to follow in one read. Safe: nothing the policy forbids. What does each reply need?

  1. Mark reply Ayes, yes, no, yes

    Four paragraphs for a two-step task fails clear.

  2. Mark reply Bno, yes, yes, yes

    The menu it names doesn't exist, so it fails correct.

  3. Total the parts3 of 4 and 3 of 4

    The totals match, just as the two 3s did.

  4. Read the failed partA: clear. B: correct.

    Each failure names its own fix.

AnswerShorten reply A. Fix the menu path in reply B.

A total adds up the parts and throws away which part failed. Kept apart, the marks say what to fix. Over many replies, each part gets its own pass rate, and you can watch the one you're working on.

Rubric

A scoring guide that splits quality into named parts, often called criteria or dimensions, and says how to mark each. Every grader, person or model, gets the same rubric.

Totals hide changes too. Over 50 replies, a prompt change kept the total at 160 of 200 marks. Underneath, correct fell from 46 to 38 while clear rose from 34 to 42. Report each part beside any total, or you'll ship a trade you never saw.

beforeaftercorrect4638complete4040clear3442safe4040Total, of 200 marks160 before, 160 after
Figure 1Pass counts for each part, out of 50 replies, before and after a prompt change. The total stays at 160 of 200 while correct falls by 8 and clear rises by 8.

Your turn

Meeting summaries are marked on four parts: accurate, complete, short and neutral. Version 1 passes 30, 45, 40 and 45 of 50. Version 2 passes 42, 45, 28 and 45. Which is true?

Both total 160. Accurate rose by 12 and short fell by 12, so the totals hide a trade. Whether it's a good one is a product call, and you can only make it if you see the parts.

Part 2. Describe every level

Count how often two graders agree, and raise it by describing each level.

A rubric only works if two graders give the same marks. Hand two support leads the same 10 replies and one instruction: "Rate correctness from 1 to 5." Then count how often their marks match.

Example 2.1

Two graders, ten replies

Grader 1 gives 4, 3, 5, 2, 4, 3, 4, 5, 3, 2. Grader 2 gives 3, 3, 4, 2, 5, 2, 4, 4, 3, 1 to the same replies, in the same order. How often do they agree?

  1. Find the matchesreplies 2, 4, 7 and 9

    Only these four get the same mark from both.

  2. Compute the agreement4 ÷ 10 = 40%

    They give the same mark to 4 replies in 10.

  3. Look at the misses6 misses, each 1 apart

    They never differ by more than one level. They see the replies alike, but each has their own idea of a 4.

Answer40%. All six disagreements are neighbours on the scale.

Grader 1Grader 21432333544225456327448549331021The same mark on replies 2, 4, 7 and 9:4 ÷ 10 = 40% agreement
Figure 2The same ten replies, marked by two graders. Matching marks are joined in orange: 4 of 10.

Without descriptions, each grader invents their own levels. One grader's 4 means "right, but wordy". The other's means "right, with one vague step". So they split even when they read a reply the same way. The fix is to write down what each level looks like.

For correctness: 5, every fact right and the steps work as written. 4, facts right, one step vague. 3, one small error that wouldn't stop the customer. 2, an error that sends the customer the wrong way. 1, wrong, off-topic or unsafe.

Anchor

The written description of one level on a scale. Anchors give every grader the same picture of a 4, so their marks match more often. The share of replies two graders mark the same is their agreement.

agreement = same marks ÷ replies = 4 ÷ 10 = 40%

Two fixes that look sensible make it worse. A 1 to 10 scale gives more levels to split over. Seeing each other's marks makes graders copy, which hides disagreement without removing it. "People as graders" corrects agreement for luck.

Your turn

Two graders mark 8 replies from 1 to 5. Grader 1: 5, 4, 4, 3, 2, 5, 1, 3. Grader 2: 5, 3, 4, 3, 3, 4, 1, 2. What's their agreement?

Replies 1, 3, 4 and 7 match: 4 ÷ 8 = 50%. Every other pair is one level apart, so counting "close" as agreeing would give 100% and hide the problem anchors fix.

Part 3. Yes or no is easier to agree on

Turn a scale into yes or no questions, and weigh what that gains and costs.

Even with anchors, the step from 3 to 4 takes judgement. Many teams ask yes or no questions instead. "Is every fact right?" has one border to argue about. A 1 to 5 scale has four: 1|2, 2|3, 3|4 and 4|5.

Example 3.1

The same marks, as pass or fail

Take the two graders' marks from part 2. Call a 4 or 5 a pass and anything lower a fail. How often do they agree now?

  1. Convert grader 1P F P F P F P P F F

    A 4 or 5 becomes P.

  2. Convert grader 2F F P F P F P P F F

    The same rule.

  3. Count the matches9 ÷ 10 = 90%

    Only reply 1 differs: a 4 against a 3.

  4. See where the misses went5 of 6 gone

    They sat at 1|2, 2|3 or 4|5, borders that pass or fail ignores.

Answer90%, up from 40%. Only the miss at the pass line is left.

123451113pass linefailpass6 disagreements; 1 crosses the pass line
Figure 3The six disagreements from part 2, each drawn across the border it straddles. A yes or no question keeps only the one at the pass line.

Every border on a scale is a place where two graders can split, so four borders give four chances to disagree. In practice you ask the yes or no question directly, such as "Does it ask for a password?", rather than squash a scale afterwards. The counting works the same way.

Yes or no check

A rubric part asked as one question with a yes or no answer, such as "Does the reply ask for a password?". It gives graders one border to agree on, where a 1 to 5 scale gives four. Also called a binary or pass/fail criterion.

The cost is detail. Nearly right and badly wrong both become no. Keep a scale for the one or two parts where that difference changes a decision, such as how wrong a medical answer is, and ask yes or no for the rest.

Your turn

Why do checklists of yes or no questions often give steadier marks than 1 to 5 scales?

A 1 to 5 scale has four borders (1|2, 2|3, 3|4, 4|5), and graders can disagree at any of them. A yes or no question has one. The cost is detail: you lose the difference between nearly right and badly wrong, so some teams keep a scale for the one or two parts where that difference matters.

Part 4. Weights and hard fails

Combine parts into one score with weights, and gate the parts you can't ship without.

To rank replies you often want one number. The usual way is a weighted sum. Each part gets a weight for how much it matters, the weights add up to 1, and a reply scores the weights of the parts it passes.

Example 4.1

Three replies, one score

Weights: correct 0.4, complete 0.3, clear 0.2, safe 0.1. Reply A fails only clear. Reply B fails only complete. Reply C fails only safe: it asks for the customer's password. Rank them.

  1. Score A0.4 + 0.3 + 0.1 = 0.8

    Every weight but clear's.

  2. Score B0.4 + 0.2 + 0.1 = 0.7

    Every weight but complete's.

  3. Score C0.4 + 0.3 + 0.2 = 0.9

    Every weight but safe's, and safe only weighs 0.1.

  4. Add a gate on safeC = 0

    Fail a gate and the reply scores 0, whatever else it passed.

AnswerWithout the gate, C ranks first at 0.9. With it: A 0.8, B 0.7, C 0.

correct.4complete.3clear.2safe.1A0.8B0.7C0.90Without a gate, C ranks first.With a gate on safe, C scores 0.
Figure 4Each bar is a reply, split by the weights of the parts. Dashed blocks are failed parts. C loses only safe's 0.1 and tops the ranking until the gate sends it to 0.

A weighted sum lets strong parts cover for a broken one. A bigger weight on safety only moves the problem: some unsafe reply will still pass enough other parts to outrank a safe one. A part you can't ship without needs a gate.

Gate

A rubric part that must pass for the reply to score at all. Fail it and the score is 0. Safety and privacy are the usual gates. The other parts are combined with weights.

score = 0 if a gate fails, else the sum of passed weights

Keep gates few. If every part is a gate, one slip anywhere zeroes an otherwise good reply, and the score stops telling good replies from bad ones.

Your turn

Weights: correct 0.5, complete 0.25, clear 0.15, safe 0.1, with no gates. Reply X fails only clear. Reply Y fails only safe. Which ranks higher?

X loses clear's 0.15 and scores 0.85. Y loses safe's 0.1 and scores 0.9. The unsafe reply ranks first because safety carries the smallest weight. A gate on safe would score Y at 0.

Where you'll meet it

In interviews

Asked how you'd grade a support bot's replies, describe a rubric. Split quality into a few parts, asked as yes or no questions with an example of each answer. Make safety a gate. Check that two graders agree before trusting anyone's marks, and report every part beside the total, so a drop points at its cause.

On the job

OpenAI's HealthBench (2025) grades replies in 5,000 health conversations against 48,562 criteria written by 262 physicians. A model grader marks each criterion met or not met. Each is worth −10 to 10 points, and harmful advice carries negative points, so a dangerous reply loses ground however much else it gets right.

A weighted rubric with a safety gate, in Python
weights = {"correct": 0.4, "complete": 0.3, "clear": 0.2, "safe": 0.1}
gates = {"safe"}
replies = {
    "A": {"correct": 1, "complete": 1, "clear": 0, "safe": 1},
    "B": {"correct": 1, "complete": 0, "clear": 1, "safe": 1},
    "C": {"correct": 1, "complete": 1, "clear": 1, "safe": 0},
}

def score(marks, use_gates=True):
    if use_gates and any(marks[g] == 0 for g in gates):
        return 0.0
    return round(sum(w * marks[k] for k, w in weights.items()), 2)

for name, marks in replies.items():
    print(name, score(marks, use_gates=False), score(marks))
# A 0.8 0.8
# B 0.7 0.7
# C 0.9 0.0

Check yourself

Four questions from across the lesson. Try each one before you look back.

1

Two graders mark the same 50 answers. Which change most likely makes their marks agree more often?

Anchors give both graders the same picture of each level, so they disagree less about where an answer sits. A 1 to 10 scale adds more levels to disagree about. Seeing each other's marks makes them copy, which hides real disagreement rather than removing it.

2

A bot passes 92 of 100 test cases against a 95% bar. With an error bar of 2 × √(p × (1 − p) ÷ n), where does 92% land?

√(0.92 × 0.08 ÷ 100) = 0.027, so two wobbles is about 5 points. The range 87% to 97% holds the 95% bar: add cases before deciding.

3

A rubric gives safety a weight of 0.1 and no gate. What can go wrong?

A reply that fails only safety keeps 0.9 of the score, more than a safe reply that fails any part weighted over 0.1. A gate would score it 0.

4

Two graders mark 20 replies from 1 to 5 and agree on 9. Turned into pass (4 or 5) or fail, the same marks agree on 18. What does the jump tell you?

On the scale they missed on 11 replies. Pass or fail only removes misses that sit inside pass or inside fail, such as a 4 against a 5. So 9 of the 11 were at those borders, and 2 straddled the pass line.

What to remember

  1. 1One score hides the reason

    Mark each part of a reply on its own: two replies can both pass 3 of 4 parts while one needs shortening and the other needs correcting.

  2. 2Describe every level

    Agreement is the share of replies two graders mark the same, 4 ÷ 10 = 40% here, and anchors for every level raise it.

  3. 3Yes or no is easier to agree on

    A yes or no question has one border to agree on and a 1 to 5 scale has four: the same marks went from 40% to 90% agreement.

  4. 4Weights and hard fails

    Gate the parts you can't ship without: with safety weighted 0.1, a reply that asked for a password scored 0.9 and ranked first.

Recap and going further

The lesson on one page

Parts, not one score
  • One number hides which part failed
  • Each part points at its own fix
  • Report every part beside the total
Agreement
  • The share of replies two graders mark the same
  • Anchors describe every level of a scale
Quality, split into parts you can mark
Yes or no
  • One border to agree on instead of four
  • Keep a scale only where degree matters
Weights and gates
  • A weighted sum ranks replies
  • Fail a gate and the reply scores 0

Try this

Mark 15 replies with your own rubric, then with a model

Pick one task, such as explaining a shop's return policy or summarising a long email. Get 15 replies from any chat assistant (ask the same thing several ways). Write a rubric of four yes or no questions, one of them a gate, and mark all 15 in a spreadsheet. Then paste the rubric and each reply into a second chat and ask it to mark them the same way.

You need: A spreadsheet and any chat assistant.

Look for

  • Your agreement with the model on each question, and whether rewording the worst one raises it.
  • Any reply that passes every question but still feels wrong, and the question that would catch it.
  • Whether the gate changes which reply ranks first.
  • How much your agreement drops if you swap one yes or no question for a 1 to 5 scale.

The words, in plain language

Rubric
A scoring guide that splits quality into named parts, often called criteria or dimensions, and says how to mark each. Every grader, person or model, gets the same rubric.
Anchor
The written description of one level on a scale. Anchors give every grader the same picture of a 4, so their marks match more often. The share of replies two graders mark the same is their agreement.
Yes or no check
A rubric part asked as one question with a yes or no answer, such as "Does the reply ask for a password?". It gives graders one border to agree on, where a 1 to 5 scale gives four. Also called a binary or pass/fail criterion.
Gate
A rubric part that must pass for the reply to score at all. Fail it and the score is 0. Safety and privacy are the usual gates. The other parts are combined with weights.

Watch

Read next

Next lessonRead the failures firstNext, where a rubric's parts come from: read real failures, sort them into piles, and measure the piles that matter. Start lesson 3

Evaluation overviewSpotted a mistake? Tell us.