Backprop through softmax and cross-entropy

Neural Networks, lesson 7

The gradient every classifier and language model starts from, derived by hand.

"Loss functions" (cross-entropy is −ln of the probability given to the right answer) and "The chain rule by hand".

Part 1. Scores into probabilities

A network that picks between more than two answers ends with one raw score per answer. Say it's guessing the next word in "a cup of ..." and the choices are tea, coffee and water. Its last layer gives scores of 2.0, 1.0 and 0.0. Scores can be any size and either sign, so they need turning into probabilities: positive, and adding up to 1.

Example 1.1

Three scores into three probabilities

tea 2.0coffee 1.0water 0.0
The raw scores.
e^2.0 = 7.389e^1.0 = 2.718e^0.0 = 1.000
Raise e (about 2.718) to each score in turn. Every result is positive, and a bigger score gives a much bigger result.
7.389 + 2.718 + 1.000=11.107
Add them up.
tea 0.665coffee 0.245water 0.090
Divide each by the total. Now they're all positive and add up to 1.

Logits

The raw scores a network gives before they're turned into probabilities. Any size, either sign.

Softmax

The recipe above: work out e^logit for each logit, then divide each by their total. Bigger logits get more of the probability.

Your turn

Add 10 to all three scores, making them 12.0, 11.0 and 10.0. What happens to the probabilities?

Adding 10 multiplies every e^score by the same e^10, and dividing by the total cancels it out. Softmax only sees the gaps between scores. That fact comes back in the last section, where it keeps the arithmetic from overflowing.

Show the answer

B. Adding 10 multiplies every e^score by the same e^10, and dividing by the total cancels it out. Softmax only sees the gaps between scores. That fact comes back in the last section, where it keeps the arithmetic from overflowing.

Part 2. The gradient on each score

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

The gradient on each score
Why it's so clean
Keeping the numbers finite

See Pro plans I have Pro: sign in

Next lessonDiagnosing a network that won't trainDead units, exploding gradients and bad starting weights, and the fixes. Start lesson 8

Neural Networks overviewSpotted a mistake? Tell us.

07 · Backprop through softmax and cross-entropy