Classical Machine Learning, lesson 1
Learning a function from examples
Features in, a label out, and a model whose numbers are fitted to data.
Machine learning starts with a table of past examples and one column you'd like to predict. This lesson names the parts of that table, and says what it means to fit a model to it. You need a little Python and school algebra: y = 2x + 1 draws a straight line.
By the end, you can:
Part 1. Features and a label
Split a table into features and a label, and spot a column that can't be a feature.
Five flats were let last month in one city. For each you have the floor area, the bedrooms, the distance from the city centre, the agent's fee and the monthly rent. A new flat is listed today, and you'd like to guess its rent before anyone signs.
Example 1.1
Five flats, one question
In this city the agent's fee is half a month's rent, paid when the lease is signed. Which columns can you use to guess the new flat's rent, and which column is the guess?
- Find the column you wantrent
It's filled in for the five past flats and blank for the new one.
- Ask when each other column is knownarea, bedrooms, distance: on listing day
They're in the advert, so the new flat has them too.
- Drop the late columnfee = rent ÷ 2
The fee only exists after signing. On old rows it gives the rent away; on the new flat it's blank.
AnswerUse area, bedrooms and distance. Rent is the column to guess. Leave the fee out.
Whatever you feed a model has to exist at the moment you use it. On listing day that's the area, bedrooms and distance, so those are the inputs, and the rent is what comes out.
Feature and label
A feature is a column you know at the moment you need a prediction, used as an input. The label is the column you want to predict: known for past rows, blank for new ones. Each row, features and label together, is one example.
area, bedrooms, distance → rent
Most late columns are harder to spot than the fee. They look like the rest, and they help in testing because they carry part of the answer. Ask of every column: would I have this on the day I use the model?
Your turn
You want to predict each flat's rent on the day it's listed. A colleague offers more columns from the agency's records. Which one can't be a feature?
Part 2. A model is a function with numbers to fill in
Write a model as a function with numbers left open, and work out what a guess predicts.
Say you expect rent to grow with area and with bedrooms. Written as a function: rent ≈ w₁ × area + w₂ × bedrooms + b. Read w₁ as "dollars for each m²", w₂ as "dollars for each bedroom", and b as a base added to every flat. The shape is your choice. The three numbers are left open.
Example 2.1
A first guess at the numbers
Guess $15 per m², $100 per bedroom and a base of $400. What does that predict for flat A (40 m², 1 bedroom, rent $1,350), flat C (60 m², 2 bedrooms, $1,700) and flat E (100 m², 3 bedrooms, $2,450)?
- Flat A15 × 40 + 100 × 1 + 400 = 1,100
The tenant pays $1,350, so the guess is $250 too low.
- Flat C15 × 60 + 100 × 2 + 400 = 1,500
$200 too low.
- Flat E15 × 100 + 100 × 3 + 400 = 2,200
$250 too low. Flats B and D, not shown, are $150 and $400 too low.
Answer$1,100, $1,500 and $2,200: every guess is too low.
The guess is a model, just a poor one. A fitting routine tries numbers until the predictions on the five flats come as close as they can to the real rents. For these flats it finds w₁ = 20, w₂ = 0 and b = 500, and every miss shrinks to $100 or less.
Model, parameters and fitting
A model is a function that turns features into a prediction. Its parameters are the numbers left open in it: one weight per feature, plus one on its own. Fitting, or training, sets them so the predictions on known examples come as close as they can to the labels.
rent ≈ w₁ × area + w₂ × bedrooms + b
The bedroom weight came out as 0 because, on these five flats, area already covers what bedrooms would add. With five rows, don't read much into that. And "as close as they can" needs a score for how wrong the numbers are: "Linear regression" builds it.
Your turn
Try another guess: $18 per m², $50 per bedroom and a base of $450. What does it predict for flat D, 80 m² with 2 bedrooms, which rents for $2,200?
Part 3. Two kinds of answer
Tell a regression task from a classification task.
Rent is a number: when the answer is $1,350, a guess of $1,360 beats a guess of $1,500. Other questions have a short list of answers: spam or not, cancel or stay, which of five teams should take a support ticket.
Example 3.1
Four questions, sorted
Which of these predict a number, and which a category? (a) Next month's sales for a shop. (b) Whether an email is spam. (c) Which of five teams should handle a support ticket. (d) How many days a flat will stay on the market.
- Next month's salesa number
$41,000 is a closer guess than $60,000 when the answer is $40,000.
- Spam or not1 of 2 categories
An email is spam or it isn't.
- Which team1 of 5 categories
Team 3 isn't "between" teams 2 and 4. A wrong team is just wrong.
- Days on the marketa number
9 days is closer than 30 when the answer is 10.
Answer(a) and (d) predict a number. (b) and (c) predict a category.
The test: does a near miss beat a far miss? If it does, you're predicting a number. If every wrong answer is just wrong, you're picking a category. A model for categories usually gives a probability for each one ("82% likely to cancel"), and you turn that into a decision.
Regression, classification and supervised learning
Regression predicts a number: a rent, a delivery time, next month's sales. Classification predicts a category from a fixed list: spam or not, one of five teams. Both are supervised learning: every training example comes with its right answer, the label, attached.
The same table can feed both kinds: "what rent will this flat fetch?" is regression, and "will it be let within two weeks?" is classification. A third kind of learning has no label at all and looks for groups among the rows. "Clustering and PCA" covers it.
Your turn
Which of these is a classification task?
Part 4. New rows are the test
Explain why a perfect score on known rows proves little about new ones.
Fitting makes a model match rows whose rents you already know. The point is the next flat. To see the difference, fit two models to the five flats and try them on three flats let the month after.
Example 4.1
A line and a curve, on new flats
The line rent = 20 × area + 500 misses the five flats by $100 or less. A curve with five numbers to set passes through all five exactly. Next month three new flats are let: 45 m² for $1,500, 70 m² for $1,850 and 90 m² for $2,350. Which model misses them by less?
- The 45 m² flatline 1,400, curve 1,323
To hit all five, the curve dips below the rent of the smaller 40 m² flat.
- The 70 m² flatline 1,900, curve 2,005
The curve swings up on its way to the 80 m² flat.
- The 90 m² flatline 2,300, curve 2,300
Both are $50 short.
- Average the missesline (100 + 50 + 50) ÷ 3 ≈ 67, curve (177 + 155 + 50) ÷ 3 ≈ 127
Perfect on the five flats, the curve misses new ones by about twice as much.
AnswerThe line: about $67 a flat, against the curve's $127.
Five rents carry a pattern (bigger flats cost more) plus noise: one landlord asked a bit more, one flat faced a busy road. A curve free enough to hit every point learns the noise too, and noise doesn't repeat. The stiffer line can't chase it, so it keeps the pattern.
Generalisation
How well a model does on new examples, not the ones it was fitted on. The only honest way to measure it is on rows the model never saw.
Range matters too. Past the rows it saw, a model is guessing. The line says $4,500 for a 200 m² flat, far bigger than any in the data, and the curve, just past the data, says $4,130 for 120 m².
Your turn
A model matches every flat it was fitted on to the dollar. What does that tell you about its error on next month's listings?
Where you'll meet it
Interviews often open with a product question: predict which customers will cancel, or how long a delivery will take. Before any model, say what one row is, what the label is, and which columns you'd really have when the prediction is made.
Tables of features and a label sit behind many everyday predictions: whether a card payment looks like fraud, when a parcel will arrive, how much stock a shop needs next week. The model is fitted again as new cases come in.
from sklearn.linear_model import LinearRegression
# Five flats. X: one row per flat, one column per feature (here just area in m²).
X = [[40], [50], [60], [80], [100]]
y = [1350, 1400, 1700, 2200, 2450] # the label: monthly rent in dollars
model = LinearRegression() # rent = w × area + b, with w and b not yet chosen
model.fit(X, y) # choose w and b to make the five predictions close
print(f"w = {model.coef_[0]:.1f} dollars per m², b = {model.intercept_:.1f} dollars")
new = [[70], [200]]
for row, rent in zip(new, model.predict(new)):
print(f"{row[0]} m²: predicted rent {rent:.0f}")
# w = 20.0 dollars per m², b = 500.0 dollars
# 70 m²: predicted rent 1900
# 200 m²: predicted rent 4500Every scikit-learn model has the same moves: make it, fit it on known rows, predict on new ones. X must be a table, one row per example, even with one feature. A flat list stops fit with "Expected 2D array, got 1D array instead", and np.array(area).reshape(-1, 1) fixes it.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A bank wants to know how much each customer will spend next month, and then to flag customers likely to spend more than $1,000. Which kinds of task are these?
2
A model rent ≈ w₁ × area + w₂ × bedrooms + b is too low on all five flats, by $150 to $400. Which single change brings every prediction closer at once?
3
A rent model was fitted on flats from 30 to 120 m². Which prediction should you trust least?
What to remember
1Features and a label
Features are the columns you know when you need the prediction, and the label is the one you want: a column filled in later, like a fee set at signing, can't be a feature.
2A model is a function with numbers to fill in
A model is a function with numbers left open: rent ≈ w₁ × area + w₂ × bedrooms + b has three parameters, and fitting set them to 20, 0 and 500.
3Two kinds of answer
If a near miss beats a far miss, it's regression; if every wrong answer is just wrong, it's classification.
4New rows are the test
Judge a model on rows it never saw: the curve that hit all five flats missed new ones by $127 on average, the line by $67.
Recap and going further
The lesson on one page
- Features: what you know when you need an answer
- Label: the column you want to predict
- A column filled in later can't be a feature
- A function with numbers left open
- rent ≈ w₁ × area + w₂ × bedrooms + b
- Set the numbers so predictions match known rows
- scikit-learn: fit(X, y), then predict(X_new)
- A number out: regression
- A category out: classification
- Both learn from labelled rows
- New rows are the test
- A curve through every known row can miss new ones badly
Try this
Fit a first model, then read a real table
In a notebook, fit LinearRegression on the five flats and predict a 70 m² flat. Add a sixth flat of your own, say 120 m² at $4,000 a month, and fit again. Then load scikit-learn's California housing data with fetch_california_housing(as_frame=True), and print its .frame.head() and its .DESCR. Decide what one row stands for, which column is the label, and what units each feature is in.
You need: Python with scikit-learn (pip install scikit-learn), or a free notebook such as Google Colab. The housing data downloads once, to your machine.
Look for
- w = 20 and b = 500 for the five flats, and how far one pricey sixth flat moves them.
- That each California row is a district (a census block group), not a single house, so a model would predict a district's typical value.
- The label's units: the description says it's in hundreds of thousands of dollars.
- Which columns you'd really know before a sale, and whether the task is regression or classification.
The words, in plain language
- Feature and label
- A feature is a column you know at the moment you need a prediction, used as an input. The label is the column you want to predict: known for past rows, blank for new ones. Each row, features and label together, is one example.
- Model, parameters and fitting
- A model is a function that turns features into a prediction. Its parameters are the numbers left open in it: one weight per feature, plus one on its own. Fitting, or training, sets them so the predictions on known examples come as close as they can to the labels.
- Regression, classification and supervised learning
- Regression predicts a number: a rent, a delivery time, next month's sales. Classification predicts a category from a fixed list: spam or not, one of five teams. Both are supervised learning: every training example comes with its right answer, the label, attached.
- Generalisation
- How well a model does on new examples, not the ones it was fitted on. The only honest way to measure it is on rows the model never saw.
Read next
- What is Machine Learning?Google, Introduction to Machine LearningSupervised and unsupervised learning, with features, labels and models used in the same sense as here.
- Getting Startedscikit-learn documentationThe fit and predict pattern that every scikit-learn model shares, on one page.
- An Introduction to Statistical LearningJames, Witten, Hastie, Tibshirani and Taylor (free PDF)Chapter 2 sets out prediction, regression versus classification, and the pull between simple and flexible models. The Python edition is free on the same site.
Classical Machine Learning overviewSpotted a mistake? Tell us.