Normalize a feature row
As numbers pass through many layers, they can drift: some rows of values become huge, others tiny, and training becomes unstable. Normalization pulls them back to a standard size. Layer normalization does this for one example at a time. It looks at the features of that example, moves them so their average is 0, and scales them so their spread is 1. Other examples in the batch play no part.
About this lesson
Words to know
- normalization
- Rescaling a group of numbers to a standard size, which keeps training stable.
- batch
- A group of examples that the model processes together in one step.
- variance
- A measure of spread: the average of the squared distances from the mean.
- epsilon
- A tiny number added before dividing or taking a root, so that the calculation is safe when the other part is 0.
Why it matters
Transformer blocks normalize their values this way, or with the lighter RMS version two lessons on, so a GPT can stack many blocks without its numbers drifting. You use it again in chapter 6.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.