Logistic regression

Classical Machine Learning, lesson 5

Turn a weighted sum into a probability, then choose where to say yes.

"Linear regression": a weighted sum of features, fitted by gradient descent to make a loss small. "Regularisation with L1 and L2" helps with the last section.

Part 1. From a yes or no to a probability

A subscription service wants to know which customers will cancel in the next 30 days. People call that churn. The label is 1 if a customer churned and 0 if they stayed. The features are things you know today: months as a customer, support tickets in the last 90 days, and whether autopay is on.

The line from "Linear regression" will fit 0s and 1s if you ask it to. Fitted to ten customers, it comes out as churn = 0.79 − 0.028 × months. After one month it says 0.76, which reads like a fair probability. At 36 months it says −0.22: a −22% chance of leaving. Fit a line to support tickets instead and you get 0.06 + 0.18 × tickets, so a customer with 8 tickets scores 1.52. Neither number can be a probability.

The fix keeps the weighted sum and adds one step at the end: bend the total into the range 0 to 1. A large positive total should come out near 1, a large negative one near 0, and a total of 0 halfway. The function that does this is 1 ÷ (1 + e^(−z)), where z is the total and e is about 2.718.

Example 1.1

One customer's churn probability

customer A3 months4 ticketsautopay off (0)
weights −0.1, 0.7, −0.9bias −1.0
Three features, a weight for each and a bias. These are the weights the data in this lesson's code was made from.
−0.1 × 3=−0.3
0.7 × 4=2.8
−0.9 × 0=0
Multiply each feature by its weight. Every month pulls the total down a little; every ticket pushes it up.
−1.0+−0.3+2.8+0=z = 1.5
Add the bias. So far this is exactly the weighted sum from "Linear regression".
1 ÷ (1 + e^(−1.5))=1 ÷ 1.223=0.818
Bend it: 1.5 becomes 0.818, an 82% chance that customer A cancels in the next 30 days.
customer B12 months, 1 ticket, autopay on
−1.0 − 1.2 + 0.7 − 0.9=−2.4→0.083
A customer of a year, with autopay on: the total is −2.4 and the probability 8%.

Sigmoid

The S-shaped function 1 ÷ (1 + e^(−z)). It maps any number into the range 0 to 1, and sends 0 to exactly 0.5. Also called the logistic function.

Logistic regression

A weighted sum of the features passed through the sigmoid, read as the probability of the yes class. Despite the name, it's used to classify, not to predict a number.

Your turn

Customer C's weighted sum, bias included, comes to exactly 0. What churn probability does the model give?

e^0 is 1, so the sigmoid gives 1 ÷ (1 + 1) = 0.5. The bias is already inside the total, so nothing is missing. A total of 0 is where the evidence for leaving and for staying balances; any positive total tips past 0.5, any negative one falls below it.

Show the answer

A. e^0 is 1, so the sigmoid gives 1 ÷ (1 + 1) = 0.5. The bias is already inside the total, so nothing is missing. A total of 0 is where the evidence for leaving and for staying balances; any positive total tips past 0.5, any negative one falls below it.

Part 2. Log loss: paying for confident mistakes

To fit the weights you need a loss: one number for how wrong the probabilities are. Mean squared error from "Linear regression" is the obvious first try, and it works badly here. It never charges more than 1 for a probability, however sure and wrong. Worse, with the sigmoid inside, a customer the model is sure about gets almost no gradient, even when it's sure of the wrong thing, so the worst mistakes are the slowest to fix.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Log loss: paying for confident mistakes
Reading the weights as odds
From a probability to a decision

See Pro plans I have Pro: sign in

Next lessonMeasuring a classifierRead a confusion matrix, and set the threshold by what mistakes cost. Start lesson 6

Classical Machine Learning overviewSpotted a mistake? Tell us.

05 · Logistic regression