insideout Get Pro

11 · Data leakage

Pro

Classical Machine Learning, lesson 11

Data leakage

Find the columns and steps that let the answer slip into training.

"Measuring a classifier" (AUC), "Cross-validation and tuning" (folds and pipelines) and "Preparing features" (steps that learn from the training rows).

Part 1. Too good to be true

Say a team trains a churn model on last year's customers and gets a cross-validated AUC of 0.99. Their previous model scored 0.78 on the same kind of data. Before anyone celebrates, someone looks at the most important column: retention_offer_made.

When a customer calls to cancel, a retention team makes an offer and ticks that box. Nearly everyone with a tick was already leaving. The model learned to read the tick, and a month before anyone calls, the column is blank for everyone.

Data leakage

Information reaching the model during training that won't be there, or won't be the same, when it makes real predictions. It inflates every score you compute and disappears in production.

Target leakage

Leakage through a feature that is filled in or changed because of the outcome: recorded after it, or by the process that handles it.

Example 1.1

One snapshot, two windows

  1. features→snapshot: 1 March→label: the next 30 days

    Pick the moment you'd make the prediction, the snapshot. Features may only use what was known by then. The label is what happens in the 30 days after.

  2. months as a customertickets in the last 90 daysdays since login
    all from before 1 March

    These can be computed from records dated before the snapshot.

  3. retention offer maderefund issuedreason for closing
    written during the label window

    These are written down while the customer is leaving. They're part of the outcome, not clues to it.

  4. tickets in the last 90 days→counted on the export date?

    Even a fair-looking column leaks if it's computed on the day the data is exported. If the export ran in May, "the last 90 days" includes the cancellation call.

Your turn

A churn table has these columns. Which one most likely leaks the label?

Customers who are leaving stop logging in, so logins after the snapshot are partly the outcome itself. The other three are fixed at the snapshot. The word to look for is "after": any count, flag or amount whose window overlaps the label's window.

Part 2. Leaks from preparing data

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Leaks from preparing data
  2. Leaks across the split
  3. Checks before you trust a score
See Pro plansI have Pro: sign in
Next lessonClustering and PCAFind groups with k-means, and squeeze many columns into a few with PCA. Start lesson 12

Classical Machine Learning overviewSpotted a mistake? Tell us.