insideout Get Pro

13 · Capstone: a tabular model end to end

Pro

Classical Machine Learning, lesson 13

Capstone: a tabular model end to end

A churn table taken from raw columns to a tested model you could ship.

"Preparing features", "Cross-validation and tuning", "Measuring a classifier" and "Data leakage". The rest of the module helps, but those four carry this lesson.

Part 1. The question and the table

This lesson takes one problem from the first question to a model you could run. The setting is the module's subscription service, which wants to keep customers from cancelling.

Start with the decision, not the data. The retention team can send a customer a $20 offer. If a customer who was about to leave gets one, say they stay, and the company keeps about $200 it would have lost. If a customer who was going to stay anyway gets one, the $20 is wasted. So the question is: each month, which customers should get the offer?

That decision sets the label. Pick a day, the snapshot date, and stand on it. For every customer active that day, the label is 1 if they cancel in the 30 days after it and 0 if they don't. The features may only use what was known on the snapshot date itself, because that's all you'll know when the model runs for real.

Example 1.1

From a decision to a table

  1. snapshot date→every customer active that day

    One row per customer. A real table stacks several snapshot dates, one per month, so the model sees different seasons.

  2. monthsbillplantickets_90d
    days_since_loginautopaycountry

    Seven features, all known on the snapshot date: months as a customer, monthly bill, plan, support tickets in the 90 days before, days since the last login, autopay on or off, and country.

  3. cancels in the next 30 days→churn = 1
    still there after 30 days→churn = 0

    The label comes from the 30 days after the snapshot. Here, 12.6% of the 5,000 customers churn.

  4. retention_call→logged when a customer phones to cancel

    One more column sits in the export: whether the retention team logged a call. It's filled in during the 30 days, after the snapshot, and mostly for customers who were already leaving.

Your turn

Before training anything, which column has to come out of the table?

retention_call is written after the snapshot, usually because the customer already asked to cancel. On the snapshot date it doesn't exist yet. Months and days since login are known that day and are exactly the signals you want; missing values and six-valued columns are things a pipeline handles, not reasons to drop a column.

This leak is loud. With retention_call in, logistic regression scores a cross-validated AUC of 0.914. Without it, 0.769. A team that kept the column would ship a model that looks excellent in every test, then finds the column empty, on the morning it runs, for the customers it's meant to catch.

Snapshot date

The day a training row pretends to stand on. Features use only what was known by then, and the label comes from a fixed window after it.

Part 2. A split, a baseline and a first honest model

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A split, a baseline and a first honest model
  2. A stronger model and a threshold, chosen on training rows
  3. One look at the test set, then nightly or live scoring
See Pro plansI have Pro: sign in
The last lesson in Classical Machine LearningBack to the overviewModels that learn from tables of data Back to Classical Machine Learning

Classical Machine Learning overviewSpotted a mistake? Tell us.