insideout Get Pro

09 · Momentum, Adam and learning-rate schedules

Pro

Neural Networks, lesson 9

Momentum, Adam and learning-rate schedules

How real training runs set the size of every weight's step.

Plain gradient descent, from "Updating weights with gradients", gives every weight the same learning rate. Real loss surfaces are steep in some directions and nearly flat in others, and one rate can't suit both. This lesson builds momentum, Adam and a changing rate on a small valley you can follow by hand.

By the end, you can:

  1. Predict when plain gradient descent zigzags, and when it crawls.
  2. Run momentum's velocity by hand, and say why β = 0.9 means about 10 steps.
  3. Work out Adam's first step, and say why it's about the learning rate.
  4. Work out a warmup-and-cosine schedule, and pick a peak learning rate.

Part 1. A narrow valley

Predict when plain gradient descent zigzags, and when it crawls.

Take two weights, x and y, and the loss 0.5·(x² + 100y²), with its floor at (0, 0). The slope is x in the x direction and 100y in the y direction. So the loss is a long, narrow trench: gentle along x, and 100 times as steep across y.

Example 1.1

Three plain steps

Start at x = 10, y = 1. Take three steps of plain gradient descent with a learning rate of 0.018: each weight minus 0.018 times its slope.

  1. Step 1x: 10 − 0.018 × 10 = 9.82; y: 1 − 0.018 × 100 = −0.8

    y jumps right over the floor to the far wall. x moves only 0.18.

  2. Step 2x: 9.82 − 0.018 × 9.82 = 9.643; y: −0.8 − 0.018 × (−80) = 0.64

    y's slope has flipped sign, so it jumps back.

  3. Step 3x: 9.470; y: −0.512

    Each step keeps 98.2% of x and turns y into −0.8 times itself.

AnswerAfter three steps, x has moved from 10 to 9.47 and y has crossed the floor three times.

Each step multiplies x by 1 − 0.018 = 0.982, and y by 1 − 0.018 × 100 = −0.8. The minus sign is the zigzag. The 0.982 is the crawl: x needs 381 steps to get within 0.01 of the floor.

plain steps, rate 0.018−1010510xystart (10, 1)floor (0, 0)after 40 steps: x = 4.84zigzag
Figure 1Forty plain steps at a rate of 0.018, drawn to scale. y zigzags across the trench and settles within a few steps. x creeps along the floor and is still at 4.84 after 40 steps.

So raise the rate? At 0.02, y's multiplier is −1 and y bounces between the walls forever; above that, each bounce grows. The steep direction sets the speed limit, and the gentle one pays for it.

Condition number

Measure a direction's steepness by how fast its slope grows: 1 per unit of x, 100 per unit of y. The condition number is the steepest over the gentlest, 100 here. Plain steps need a rate below 2 ÷ the steepest, so the gentlest shrinks by at most 2% a step.

each step: direction × (1 − rate × steepness)

A smaller rate cures the zigzag but not the crawl: at 0.01, y lands on the floor in one step, but x needs 688. A network's loss has a direction for every weight, and some are far steeper than others.

Your turn

A different valley: 0.5·(x² + 25y²). What learning rate must plain gradient descent stay below, and how much of x is left after each step near that limit?

y is multiplied by 1 − 25 × rate, which stays between −1 and 1 only below 2 ÷ 25 = 0.08. Near that, x is multiplied by 1 − 0.08 = 0.92. At 0.04, y lands on the floor in one step, but that's half the limit.

Part 2. Momentum, a running velocity

Run momentum's velocity by hand, and say why β = 0.9 means about 10 steps.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Momentum, a running velocity
  2. A step size for each weight
  3. Warmup and cosine decay
See Pro plansI have Pro: sign in
Next lessonTrain a classifier on your own autogradPut the pieces together and train a small network end to end. Start lesson 10

Neural Networks overviewSpotted a mistake? Tell us.