Normalize by root mean square
RMS normalization is a lighter version of layer normalization, and many recent language models use it. It skips the step of subtracting the mean. It only measures how large the values in a row are, with a number called the root mean square, and divides by it. Rows of any size come out at a standard size, and the work per row is smaller.
About this lesson
Words to know
- normalization
- Rescaling a group of numbers to a standard size, which keeps training stable.
- root mean square
- A measure of size: square the numbers, take their mean, then take the square root.
- epsilon
- A tiny number added before dividing or taking a root, so that the calculation is safe when the other part is 0.
Why it matters
Several recent open language models, Llama among them, use RMS normalization in place of layer normalization inside each transformer block. It does the same job with less arithmetic.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.