Production AI, lesson 14
Feature stores and training-serving skew
Compute each feature the same way for training and for live traffic.
A model trains on features that one piece of code computed, and once it's live, another piece of code computes them. When the two disagree, the model quietly gets worse and nothing raises an error. You need to know what a feature and a training set are.
By the end, you can:
- Spot a feature that training and live code compute differently, and say what it does to the model.
- Give each training row the feature values known at its own time, and nothing later.
- Decide when shared functions are enough and when a team needs a feature store.
- Compare the features a live model saw with what training code computes for the same requests, and read the pattern.
Part 1. One feature, computed twice
Spot a feature that training and live code compute differently, and say what it does to the model.
In the last lesson, cheap gates made calls about each ticket, such as whether it's urgent. A number about the customer helps: how many tickets they've sent in the last 7 days. A third ticket this week usually means a more upset customer than a first.
Two pieces of code compute that number. For training, a nightly query counts each old ticket's earlier tickets in the warehouse. For a live ticket, the ticket service counts the customer's tickets in its own database, after saving the new one.
Example 1.1
One ticket, two counts
Customer 42 writes on Monday, her first ticket in a month, and again on Wednesday. What does each piece of code give Wednesday's ticket for "tickets in the last 7 days"?
- Count it the training way1
The query counts tickets before this one, and only Monday's qualifies.
- Count it the live way1 + 1 = 2
The service saved Wednesday's ticket before counting, so it counts Monday's and this one.
- Read 2 as the model does2 earlier tickets
In every training row, 2 meant two earlier tickets. So the model treats her second contact like a third.
AnswerTraining gives 1 and the live service gives 2. Every live ticket looks like one more contact than it is.
Nothing crashes. Each value is a believable number, one too high, on every live ticket. A gate that learned that repeat contacts are urgent now sends more tickets to the urgent queue than it did in testing. Its test scores still look fine, because they were measured on training's counts.
Training-serving skew
A difference between the feature values a model trained on and the ones it gets live, for the same case. The usual cause is one feature computed by two pieces of code.
Off-by-one counts are one kind. Others: a day that starts at midnight UTC on one side and local midnight on the other, seconds against milliseconds, a missing value filled with the average in training and 0 live, a library upgraded on one side only.
Your turn
Customer 18 writes on a Tuesday. Her two earlier tickets were sent 10 and 12 days ago. What do training and the live service give this ticket for "tickets in the last 7 days"?
Part 2. Only what was known then
Give each training row the feature values known at its own time, and nothing later.
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Only what was known then
- Offline and online stores
- Catching skew in production
Production AI overviewSpotted a mistake? Tell us.