insideout Get Pro

12 · Clustering and PCA

Pro

Classical Machine Learning, lesson 12

Clustering and PCA

Find groups with k-means, and squeeze many columns into a few with PCA.

"Learning a function from examples" (features and labels), "Linear regression" (squared error) and "Preparing features" (standardisation).

Part 1. Data with no label

Sometimes there's no label. A product team wants to know what kinds of customers they have, to design plans for them, and nobody has written a type next to each customer. Do the rows fall into groups on their own?

Unsupervised learning

Learning from features alone, with no label to predict. The two common jobs are finding groups of similar rows (clustering) and finding a shorter description of the columns, as PCA does.

The most used clustering method, k-means, is short enough to run by hand. Choose how many groups you want, k, and place k centres somewhere. Then repeat two moves: give each row to its nearest centre, and move each centre to the average of its rows. Here it is on six customers' monthly bills with k = 2, started badly on purpose, at $10 and $20.

Example 1.1

k-means on six bills

  1. bills$10$14$20$60$70$86
    centres1020

    Six bills and two starting centres, both at the low end.

  2. assign10, 1420, 60, 70, 86
    total squared distance8,472

    Each bill goes to its nearest centre. $14 is 4 from the first and 6 from the second. Everything from $20 up goes to the second.

  3. move1259
    total2,380

    Each centre moves to the average of its bills: (10 + 14) ÷ 2 = 12, and (20 + 60 + 70 + 86) ÷ 4 = 59.

  4. assign10, 14, 2060, 70, 86
    total923

    $20 is now 8 from 12 and 39 from 59, so it switches sides.

  5. move14.6772
    total394.67

    The centres move again. Assigning once more changes nothing, so k-means stops with a low group and a high group.

K-means

Clustering that repeats two steps until nothing changes: assign each row to the nearest of k centres, then move each centre to the mean of its rows. You choose k.

Inertia

The total squared distance from each row to its centre, k-means' measure of how tight the groups are. Lower is tighter. scikit-learn reports it as inertia_.

Your turn

The totals after each step were 8,472, 2,380, 923 and 394.67. Could an assign or a move step ever make the total go up?

Giving a row to its nearest centre can't increase its distance, and a group's mean is the point with the smallest total squared distance to its members. So the total never rises, and since there are only so many ways to assign rows, k-means always stops. Where it stops depends on where it started.

The lab runs the same two moves on points in two dimensions. Press Assign and Move centres in turn and watch the total fall. Then press New start a few times with the same k, and compare where each run ends.

Step through k-meansLive

Part 2. Choosing k, and where k-means goes wrong

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Choosing k, and where k-means goes wrong
  2. PCA: fewer columns, most of the spread
  3. When you'd use them
See Pro plansI have Pro: sign in
Next lessonCapstone: a tabular model end to endA churn table taken from raw columns to a tested model you could ship. Start lesson 13

Classical Machine Learning overviewSpotted a mistake? Tell us.