Differentiate one neuron
A neuron gives an answer, and the answer is off. Which way should its weight move, and by how much? Backpropagation answers that by walking backwards through the calculation. At each step it asks one small question: if this value went up a little, how much would the next value change? Multiplying those small answers together gives the effect of the weight on the loss. That product is the gradient.
About this lesson
Words to know
- backpropagation
- Working out the gradient of every weight by going backwards from the loss, one step of the calculation at a time.
- chain rule
- If A changes B and B changes C, the effect of A on C is the two effects multiplied together.
- gradient
- The slope of the loss for one number in the model: which way the loss moves, and how fast, when that number goes up a little.
Why it matters
A deep network repeats this for every weight in every layer. PyTorch does it when you call loss.backward(), and chapter 4 relies on that.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.