Backpropagate through two layers
Real networks have more than one layer, so the error has to travel back through each of them. The weight in the first layer does not touch the output directly. It changes a hidden value, and the hidden value changes the output. So its gradient is the output's error, passed back through everything in between. One of those things is a ReLU, which can block the path completely.
About this lesson
Words to know
- hidden layer
- A layer of neurons between the input and the output. Its values are not seen from outside.
- chain rule
- If A changes B and B changes C, the effect of A on C is the two effects multiplied together.
- ReLU
- The function
max(0, x). It keeps positive numbers and turns everything else into 0. - preactivation
- The value that goes into an activation function: the neuron's weighted sum plus bias, before the bend.
Why it matters
The same rule carries the error back through every layer of a deep network. Units whose ReLU blocks the path come back in chapter 4 as dead units.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.