Scale initial weights
Before training, every weight needs a starting value. If all weights start equal, every neuron in a layer computes the same thing and they all learn the same thing, so the layer is no better than one neuron. Random starting values fix that. But their size matters too: too large and the numbers flowing through the network blow up, too small and they fade to nothing. He initialization picks a size that keeps them steady through ReLU layers.
About this lesson
Words to know
- fan-in
- The number of inputs that feed one neuron.
- standard deviation
- A measure of how spread out numbers are: the square root of the variance.
- ReLU
- The function
max(0, x). It keeps positive numbers and turns everything else into 0.
Why it matters
PyTorch layers pick sensible starting weights by a rule like this one. When a deep network refuses to learn from the first step, the starting scale is one of the first things to check.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.