insideout Get Pro

06 · Linear regression

Pro

Statistics and Regression, lesson 6

Linear regression

Fitting the best line by hand, then by formula.

Correlation says how closely GULL follows the market, and a regression line says how far: for each 1% the market moves, how much does GULL move? This lesson fits that line by hand, then by formula. You need correlation and standard deviations from the last lessons.

By the end, you can:

  1. Score a line by its squared misses.
  2. Compute the least-squares slope and intercept from sums.
  3. Measure the share of y's variance a line explains.
  4. Work out the slope of the reverse regression.

Part 1. The line with the smallest misses

Score a line by its squared misses.

Take the five weeks of GULL and the market from the last lesson. Any line through them predicts GULL's return from the market's, and misses each week by some amount. To call one line the best, you need a single score for all five misses.

Example 1.1

Two lines through five weeks

The market returns +1%, −1%, +3%, 0% and +2%, and GULL 0%, −1%, +2%, +1% and +3%. Line A predicts GULL = market. Line B predicts GULL = 0.2 + 0.8 × market. Which misses less?

  1. Line A's misses−1, 0, −1, 1, 1

    Each week, GULL's return minus the line's. They add to 0.

  2. Line B's misses−1, −0.4, −0.6, 0.8, 1.2

    For the first week, 0 − (0.2 + 0.8 × 1) = −1. They add to 0 too.

  3. Square and add, line A1 + 0 + 1 + 1 + 1 = 4

    Squares can't cancel.

  4. Square and add, line B1, 0.16, 0.36, 0.64, 1.44

    They add to 3.6, less than 4.

AnswerLine B, with squared misses of 3.6 against 4.

Both lists of misses add to exactly 0, so adding can't tell the lines apart. Any line through the point of averages, here 1% and 1%, does that, even a flat one. Squaring makes every miss count, and big misses count most.

−1%−1%0%0%+1%+1%+2%+2%+3%+3%0.2 + 0.8 × marketsquared misses: 3.6no other line does betterthe market's weekly return, across; GULL's, up
Figure 1Line B through the five weeks, with each miss, measured up or down, drawn as a square. Their areas add to 3.6, and no other line has a smaller total.

Residual and least squares

A point's residual is its actual y minus the line's y: how far it sits above or below the line. For a line a + bx, least squares picks the a and b with the smallest sum of squared residuals. Fitting a line this way is linear regression, often called OLS, for ordinary least squares.

Σ(y − a − bx)² as small as it can be

Squaring has a cost. A miss twice as big counts four times as much, so one far point can drag the whole line towards itself, as one crash week moved r in the last lesson. Adding absolute misses instead resists that, but it has no simple formula.

Your turn

On the same five weeks, line C predicts GULL = 0.5 + 0.5 × market. What's its sum of squared residuals?

Its predictions are 1, 0, 2, 0.5 and 1.5, so the residuals are −1, −1, 0, 0.5 and 1.5. They add to 0, which is why adding can't score a line. Their squares add to 1 + 1 + 0 + 0.25 + 2.25 = 4.5, worse than line B's 3.6. 2.1 is √4.5.

Part 2. The formula for the best line

Compute the least-squares slope and intercept from sums.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. The formula for the best line
  2. How much the line explains
  3. y on x isn't x on y
See Pro plansI have Pro: sign in
Next lessonWhat regression assumesErrors, outliers and the assumptions that quietly break. Start lesson 7

Statistics and Regression overviewSpotted a mistake? Tell us.