insideout Get Pro

02 · Linear regression

Free

Classical Machine Learning, lesson 2

Linear regression

Fit a line by mean squared error, with gradient steps or one formula.

Fitting needs a score for how wrong a line is. This lesson builds that score, then finds the line that makes it smallest in two ways: by small steps downhill, and by one formula. You need the five flats from "Learning a function from examples": features, a label and a model with numbers to fill in.

By the end, you can:

  1. Score a line by its mean squared error.
  2. Take a gradient-descent step, and pick a learning rate that neither stalls nor explodes.
  3. Find the least-squares line directly, with no steps.
  4. Read a coefficient and R², without reading too much into them.

Part 1. Mean squared error scores a line

Score a line by its mean squared error.

Keep one feature of the five flats, the area: 40 m² for $1,350, 50 for $1,400, 60 for $1,700, 80 for $2,200 and 100 for $2,450. A line through them is rent = w × area + b, with a slope w in dollars per m² and an intercept b. Try w = 25 and b = 200.

Example 1.1

Scoring rent = 25 × area + 200

How far is the line rent = 25 × area + 200 from the five flats, in one number?

  1. Predict each flat1,200, 1,450, 1,700, 2,200, 2,700

    25 × 40 + 200 = 1,200, and so on.

  2. Take actual minus predicted150, −50, 0, 0, −250

    Positive means the line runs below that flat.

  3. Square each gap22,500 + 2,500 + 0 + 0 + 62,500 = 87,500

    The signs disappear, and the big miss now makes up most of the total.

  4. Average87,500 ÷ 5 = 17,500

    One number for the whole line. Lower is better.

Answer17,500, a typical miss of √17,500 ≈ $132.

1,2001,6002,0002,400406080100area, m²rent $+150−50−25025 × area + 200
Figure 1Each stick is one flat's gap from the line. The −250 at 100 m², in orange, is 62,500 of the 87,500 once squared.

Why square? The gaps add up to −150, so their plain average is −30, which looks nearly perfect because misses in opposite directions cancel. Squaring stops that, makes one big miss cost more than several small ones, and gives a smooth score with an exact formula for its lowest point.

Residual and mean squared error

A residual is actual minus predicted, for one row. The mean squared error, MSE, is the average of the squared residuals. Its square root is back in the label's units: a typical miss.

MSE = (r₁² + r₂² + … + rₙ²) ÷ n

The line rent = 20 × area + 500 leaves residuals of 50, −100, 0, 100 and −50: an MSE of 5,000, a typical miss of about $71. No line does better on these flats, as part 3 shows. If big misses cost you no extra, average the sizes of the misses instead: the mean absolute error.

Your turn

Line P misses one flat by $300 and gets the other four exactly right. Line Q misses all five flats by $60 each. Which has the higher mean squared error?

P: 300² ÷ 5 = 18,000. Q: 60² × 5 ÷ 5 = 3,600. Both miss by $300 in total, so the mean absolute error calls them equal at $60. Squaring charges P five times as much, because one $300 miss counts for far more than five $60 ones.

Part 2. Gradient descent walks downhill

Take a gradient-descent step, and pick a learning rate that neither stalls nor explodes.

Every pair (w, b) has its own MSE, and for squared error they form a bowl with the best line at the bottom. The gradient says which way is downhill: how fast the MSE changes as you nudge each number. For b it's −2 × the average residual. For w it's −2 × the average of area × residual.

Example 2.1

One step from w = 25, b = 200

The line rent = 25 × area + 200 leaves residuals of 150, −50, 0, 0 and −250. Take one step, with a learning rate of 0.0001.

  1. Gradient for b−2 × (−150 ÷ 5) = 60

    The residuals add up to −150: overall the line is too high, so b should come down.

  2. Gradient for w−2 × (−21,500 ÷ 5) = 8,600

    Area × residual is 6,000, −2,500, 0, 0 and −25,000. The 100 m² flat dominates.

  3. Step against bothw: 25 − 0.0001 × 8,600 = 24.14, b: 200 − 0.0001 × 60 = 199.994

    Each number takes a step the opposite way to its gradient, sized by the learning rate.

  4. Score the new line17,500 → 13,669

    Then repeat from the new line.

Answerw = 24.14 and b = 199.994, and the MSE falls from 17,500 to 13,669.

Gradient descent and learning rate

Gradient descent fits a model by repeated small steps: work out the gradient of the loss for every parameter, move each one a little against it, and repeat until the loss stops falling. The learning rate scales each step.

new w = w − learning rate × gradient for w

Look at the two moves: w by 0.86, b by 0.006. w's gradient multiplies every residual by an area of 40 to 100, so it's far bigger. At this rate b crawls: about 177,000 steps to get within $10 of 500. Ten times the rate, and w overshoots further each step until the MSE explodes.

The fix is to rescale the area: subtract its average, 66, and divide by its spread, about 21.5. Both gradients then come out the same size, and a rate of 0.2 suits both. "Preparing features" covers rescaling.

05,00010,00015,00020,00005101520stepMSEm², rate 0.001: 300,092 after one step, then worsem², rate 0.0001: stalls at 13,660rescaled, rate 0.2: 5,000
Figure 2Three runs from the line 25 × area + 200. In m², rate 0.0001 stalls and rate 0.001 explodes. With area rescaled, rate 0.2 reaches the best MSE, 5,000, within 10 steps.

Your turn

Start instead from w = 15 and b = 500. The residuals are 250, 150, 300, 500 and 450, and area × residual adds up to 120,500. With a learning rate of 0.0001, where does w go?

The gradient for w is −2 × (120,500 ÷ 5) = −48,200. It's negative, so w goes up: 15 − 0.0001 × (−48,200) = 19.82, close to the best 20. 10.18 steps the wrong way, and 39.10 uses the sum instead of the average.

Part 3. One formula finds the best line

Find the least-squares line directly, with no steps.

For squared error, the bottom of the bowl can be found exactly. Measure each flat's gap from the average flat: dx for area and dy for rent. The slope is the sum of dx × dy over the sum of dx × dx. The intercept then puts the line through the average flat.

Example 3.1

The best line for the five flats

Find the slope and intercept that give the five flats the lowest possible MSE.

  1. Average each columnarea 66, rent 1,820

    The areas add up to 330 and the rents to 9,100: 330 ÷ 5 = 66 and 9,100 ÷ 5 = 1,820.

  2. Measure the gapsdx: −26, −16, −6, 14, 34, dy: −470, −420, −120, 380, 630

    Small flats sit below both averages, big flats above.

  3. Add up dx × dyΣ(dx × dy) = 46,400

    The products are 12,220, 6,720, 720, 5,320 and 21,420, all positive: each flat sits on the same side of both averages.

  4. Add up dx²Σ(dx²) = 2,320

    676, 256, 36, 196 and 1,156 added up: how spread out the areas are.

  5. Divide, then place the line46,400 ÷ 2,320 = 20, 1,820 − 20 × 66 = 500

    The intercept makes the line pass through the average flat.

Answerrent = 20 × area + 500, the line gradient descent was heading for.

1,2001,6002,0002,400406080100area, m²rent $dx = 34dy = 630average flat66 m², $1,82020 × area + 500
Figure 3The best line passes through the average flat. For the 100 m² flat, dx = 34 and dy = 630: on its own, a slope of 630 ÷ 34 ≈ 18.5.

Each flat suggests a slope of its own, dy ÷ dx: for the 100 m² flat, 630 ÷ 34 ≈ 18.5. The formula averages these suggestions and gives flats far from the average area more say, since noise moves their slopes less.

Least squares and closed form

Least squares picks the parameters with the smallest sum of squared residuals, which is also the smallest MSE. For a line the answer has a closed form: a formula you compute directly, with no steps.

slope = Σ(dx × dy) ÷ Σ(dx²), intercept = average rent − slope × average area

With several features, the best weights solve the normal equations, (XᵀX) w = Xᵀy, where X has one row per flat plus a column of 1s. That gets slow with tens of thousands of features, and most losses have no formula, so most models, neural networks included, are fitted by steps.

Your turn

Fit a line to four points: (0, 1), (1, 3), (2, 2) and (3, 6). What are the least-squares slope and intercept?

The averages are 1.5 and 3, so Σ(dx × dy) is 3 + 0 − 0.5 + 4.5 = 7, and Σ(dx²) is 5. The slope is 7 ÷ 5 = 1.4, and the intercept 3 − 1.4 × 1.5 = 0.9. Slope 1.67 joins the first and last points and ignores the middle two.

Part 4. Reading the fitted line

Read a coefficient and R², without reading too much into them.

In rent = 20 × area + 500, the 20 says each extra m² goes with $20 more a month. Its units are the label's over the feature's: per square foot (1 m² ≈ 10.76 sq ft) it's $1.86, with identical predictions. To judge the whole fit, compare it with always guessing the average rent.

Example 4.1

How much of the spread the line explains

How much better is rent = 20 × area + 500 than predicting the average rent, $1,820, for every flat?

  1. Spread before the model470² + 420² + 120² + 380² + 630² = 953,000

    The squared gaps of the rents from their average: what always guessing $1,820 leaves.

  2. Spread after the model50² + 100² + 0² + 100² + 50² = 25,000

    The squared residuals of the fitted line.

  3. Take the share explained1 − 25,000 ÷ 953,000 ≈ 0.974

    The line removes 97.4% of the squared spread.

AnswerR² ≈ 0.974.

Coefficient and R²

A coefficient is the weight on one feature: how much the prediction changes when that feature rises by 1 and the others stay the same. R² is the share of the label's spread around its average that the model explains: 1 is perfect, 0 is no better than the average.

R² = 1 − (sum of squared residuals ÷ sum of squared gaps from the average)

A coefficient describes a pattern, not a cause. Buildings with a concierge sit on dear streets, so a rent model may put a big weight on "concierge", but hiring one wouldn't add that much to the rent. The same goes for any feature you could change.

R² on the rows a model was fitted on creeps towards 1 as you add features, even random ones, so measure it on new rows. There it can even drop below 0, when the model does worse than guessing the new rows' own average.

Your turn

On five new flats, a model's squared residuals add up to 60,000. The squared gaps of those rents from their own average add up to 50,000. What is the model's R² on them?

60,000 ÷ 50,000 = 1.2, and 1 − 1.2 = −0.2. Below 0: on these flats, the model does worse than guessing their average. 0.83 divides the wrong way round, and 0.2 drops the sign.

Where you'll meet it

In interviews

A common screen: derive the gradient of the MSE for a line, then say when you'd use gradient descent rather than the formula. The formula is exact but slows down with very many features. Steps scale to big data and work for losses with no formula.

On the job

scikit-learn's LinearRegression is plain least squares, solved by SciPy's lstsq, which works on X itself rather than XᵀX and so stays accurate when two columns are near copies. For more than about 10,000 training rows, scikit-learn's guide suggests SGDRegressor, which fits by gradient steps.

Three ways to the same line, in Python
import numpy as np
from sklearn.linear_model import LinearRegression

area = np.array([40, 50, 60, 80, 100], dtype=float)
rent = np.array([1350, 1400, 1700, 2200, 2450], dtype=float)

# 1. The formula: slope from the gaps to the averages, then the intercept
dx, dy = area - area.mean(), rent - rent.mean()
slope = (dx * dy).sum() / (dx * dx).sum()
print(f"formula:  slope {slope:.2f}, intercept {rent.mean() - slope * area.mean():.2f}")

# 2. scikit-learn's least-squares solver
model = LinearRegression().fit(area.reshape(-1, 1), rent)
print(f"sklearn:  slope {model.coef_[0]:.2f}, intercept {model.intercept_:.2f}")

# 3. Gradient descent on rescaled area, 20 steps at learning rate 0.2
z = dx / area.std()               # area minus 66, divided by about 21.5
a, c = 0.0, 0.0                   # rent = a × z + c
for step in range(20):
    err = rent - (a * z + c)
    a += 0.2 * 2 * (err * z).mean()   # step against the gradient
    c += 0.2 * 2 * err.mean()
w = a / area.std()                 # back to dollars per m²
print(f"descent:  slope {w:.2f}, intercept {c - w * area.mean():.2f}")

mse = ((rent - model.predict(area.reshape(-1, 1))) ** 2).mean()
print(f"MSE {mse:.0f}, R² {1 - mse / rent.var():.3f}")
# formula:  slope 20.00, intercept 500.00
# sklearn:  slope 20.00, intercept 500.00
# descent:  slope 20.00, intercept 499.98
# MSE 5000, R² 0.974

The formula and scikit-learn agree to the cent. Twenty gradient steps on rescaled area, starting from 0, stop 2 cents short of $500. The last line prints the MSE from part 1 and the R² from part 4.

Check yourself

Three questions from across the lesson. Try each one before you look back.

1

You fit rent = w × area + b by gradient descent with area in m², and at a learning rate of 0.0001 the intercept b barely moves. To hurry it along, you raise the rate tenfold, to 0.001. What happens?

The first step, 0.001 × 8,600 = 8.6, throws w from 25 to 16.4, further past 20 than it started. Each step overshoots more than the last: w goes 16.4, 90.7, −551, and the MSE passes 21 million in two steps. In raw m², any rate above about 0.0002 blows up.

2

A rent model fitted on area and bedrooms gives 22 dollars per m² and −60 dollars per bedroom. What does the −60 tell you?

Flats with more bedrooms are usually bigger, and the area term carries that. The −60 compares flats of the same size: an extra bedroom means smaller rooms, which in this data went with slightly lower rent. It says nothing about what putting up a new wall would do.

3

A least-squares line fitted to the five flats always passes through one particular point. Which?

The intercept is average rent minus slope × average area, so at the average area the line gives exactly the average rent: 20 × 66 + 500 = 1,820. It needn't touch any real flat.

What to remember

  1. 1Mean squared error scores a line

    MSE is the average squared residual: 17,500 for the line 25 × area + 200, and 5,000 for the best line, a typical miss of $71.

  2. 2Gradient descent walks downhill

    Gradient descent steps each number the opposite way to its gradient, sized by the learning rate: in raw m², any rate above about 0.0002 explodes, so rescale features first.

  3. 3One formula finds the best line

    Least squares has a closed form: the slope is Σ(dx × dy) ÷ Σ(dx²), here 46,400 ÷ 2,320 = 20, and the line passes through the average point.

  4. 4Reading the fitted line

    A coefficient is a pattern with the other features held fixed, not a cause, and R² = 1 − 25,000 ÷ 953,000 ≈ 0.974 is the share of spread explained.

Recap and going further

The lesson on one page

Residuals
  • Actual minus predicted, one per row
  • Positive: the line runs below that row
Mean squared error
  • Square each residual, then average
  • One big miss costs more than several small ones
Gradient descent
  • Move each number a little against its gradient, repeat
  • Too big a learning rate and it blows up
Fit a line: score it, then step or solve
Scale matters
  • Raw m² make the slope's gradient huge and the intercept crawl
  • Rescale the feature and both move together
Closed form
  • slope = Σ(dx × dy) ÷ Σ(dx²)
  • The line passes through the average point
Reading the fit
  • $20 per extra m², others held fixed
  • Coefficients describe data, not causes
  • R² 0.974: share of the spread explained

Try this

Watch gradient descent stall, explode, then settle

In a notebook, run the gradient-descent loop on the five flats twice over. First on raw area in m², from w = 25 and b = 200, with learning rates 0.0001, 0.0002 and 0.00021. Then on rescaled area with 0.2 and 1.1. Record the MSE at every step and plot all five runs on one chart with matplotlib (put the MSE axis on a log scale). Last, check the answer with np.linalg.lstsq on a table that has a column of 1s next to the areas.

You need: Python with numpy and matplotlib (pip install numpy matplotlib), or a free notebook such as Google Colab.

Look for

  • The raw-unit runs, where the slope settles quickly and the intercept hardly moves.
  • The learning rate where raw units tip from slow to exploding, near 0.000207.
  • The rescaled run at 0.2 reaching an MSE of 5,000 within twenty steps.
  • The same 20 and 500 from lstsq.

The words, in plain language

Residual and mean squared error
A residual is actual minus predicted, for one row. The mean squared error, MSE, is the average of the squared residuals. Its square root is back in the label's units: a typical miss.
Gradient descent and learning rate
Gradient descent fits a model by repeated small steps: work out the gradient of the loss for every parameter, move each one a little against it, and repeat until the loss stops falling. The learning rate scales each step.
Least squares and closed form
Least squares picks the parameters with the smallest sum of squared residuals, which is also the smallest MSE. For a line the answer has a closed form: a formula you compute directly, with no steps.
Coefficient and R²
A coefficient is the weight on one feature: how much the prediction changes when that feature rises by 1 and the others stay the same. R² is the share of the label's spread around its average that the model explains: 1 is perfect, 0 is no better than the average.

Watch

Read next

Next lessonTraining, validation and test setsNext, why a low error on the rows you fitted can fool you, and how to keep rows aside for an honest score. Start lesson 3

Classical Machine Learning overviewSpotted a mistake? Tell us.