Scoring generated text

Evaluation, lesson 15

Fluency, faithfulness and relevance, and which metric fits which.

"Rubrics", "Overlap metrics: BLEU, ROUGE, BERTScore" and "Models as graders": three ways to score an output. This lesson sorts out which fits which kind of quality.

Part 1. Four ways a summary can fail

Here's a short news item: "The city council voted 7 to 2 on Tuesday to close Elm Street to cars on weekends from June. Shop owners on Elm Street asked for the change. The council will review it after six months." Now look at four bad summaries of it.

Example 1.1

Four different failures

"Council Elm Street close weekend voted cars June the."
Reads badly. The words are roughly right, but it isn't proper English.
"Shop owners asked for it. The vote was 7 to 2. It closes on weekends. Elm Street is a street."
Each sentence is fine, but they don't hang together: the main point comes last and something is padding.
"The council unanimously voted to close Elm Street to cars on weekends from June."
Smooth and clear, and wrong: the vote was 7 to 2, not unanimous. It states something the source doesn't support.
"The council meets on Tuesdays and has nine members."
Well written and arguably true, but it misses what the story was about.

Fluency, coherence, factuality, relevance

Fluency: does each sentence read as natural language? Coherence: do the sentences fit together in a sensible order? Factuality (or faithfulness, when judged against a source): is every claim supported? Relevance: does it cover what matters for the task? SummEval, a widely used study of summary scoring, had expert raters score news summaries on these four, calling factuality "consistency".

Your turn

Against a reference summary "The council voted 7 to 2 to close Elm Street to cars on weekends from June", which bad summary would ROUGE score highest?

It shares almost every word with the reference, and swapping "7 to 2" for "unanimously" costs only a few words. It's also the most dangerous failure, because it reads perfectly. Overlap scores are weakest exactly where errors are hardest to spot.

Show the answer

C. It shares almost every word with the reference, and swapping "7 to 2" for "unanimously" costs only a few words. It's also the most dangerous failure, because it reads perfectly. Overlap scores are weakest exactly where errors are hardest to spot.

Part 2. Checking facts one claim at a time

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Checking facts one claim at a time
Match the measure to the goal
When there's no single right answer

See Pro plans I have Pro: sign in

Next lessonCapstone: an evaluation reportEvaluate a real system and write up what you found. Start lesson 16

Evaluation overviewSpotted a mistake? Tell us.

15 · Scoring generated text