insideout Get Pro

05 · The chain rule by hand

Pro

Neural Networks, lesson 5

The chain rule by hand

Work out each weight's gradient through a small network, step by step.

"Updating weights with gradients": training moves each weight against its slope, and you can estimate a slope by nudging the weight a hair.

Part 1. Rates multiply

A bakery turns flour into loaves and loaves into money. Each extra kilo of flour makes 4 more loaves. Each loaf sells for 3. So each extra kilo of flour brings in 4 × 3 = 12 more.

That's the whole idea behind backpropagation. When one quantity feeds the next, and that one feeds the next, the overall rate is the rates along the way multiplied together.

Your turn

Each extra kilo of flour makes 4 more loaves, and each loaf sells for 3. A quarter of every sale goes on rent. How much more rent does each extra kilo of flour bring in?

Flour to loaves is 4, loaves to sales is 3, sales to rent is 0.25. Multiply along the chain: 4 × 3 × 0.25 = 3. Multiplying is what carries a change from one step to the next; adding the rates mixes up loaves, money and rent.

Chain rule

When a change passes through several steps, the overall rate is the product of each step's rate. In calculus these rates are called derivatives.

One piece of notation, because papers and docs use it everywhere. The rate at which the loss L changes when a weight w moves is written ∂L/∂w, and read "the slope of L with respect to w". The curly ∂ is a reminder that every other weight is held still while w moves.

Part 2. A tiny network, forward

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A tiny network, forward
  2. Backward, one link at a time
  3. Check it by nudging
See Pro plansI have Pro: sign in
Next lessonAutograd you can step throughWrite backward() for a few operations and watch gradients flow through a graph. Start lesson 6

Neural Networks overviewSpotted a mistake? Tell us.