Scoring agents

Agents, lesson 11

Task success, tool choice, cost and number of steps.

You know what a trace records ("Traces you can debug from"). Evaluation covers scoring in general; this lesson is about what's different for agents.

Part 1. Check the world, not the reply

Task: "Change the delivery address on order 5521 to 14 Hill Road." The agent replies "Done! The address is updated." A grader that reads that reply would mark it a success. A grader that looks up order 5521 in the database might find the old address still there, or the new address on order 5512.

Your turn

What's the most trustworthy way to score this task?

Agents act on the world, so check the world. A code check on the final state can't be talked into a pass. Judging the reply (by a model or a keyword) scores how the agent describes its work, which is exactly where agents go wrong.

Show the answer

D. Agents act on the world, so check the world. A code check on the final state can't be talked into a pass. Judging the reply (by a model or a keyword) scores how the agent describes its work, which is exactly where agents go wrong.

Task completion

Whether the run left the world in the state the task asked for, checked by code after the run: a database row, a passing test suite, a file with the right contents.

Part 2. Run each task more than once

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Run each task more than once
The path matters too
A suite you rerun on every change

See Pro plans I have Pro: sign in

Next lessonSeveral agents on one taskWhen splitting work across agents helps, and what it costs. Start lesson 12

Agents overviewSpotted a mistake? Tell us.

11 · Scoring agents