Normalize a training batch
Batch normalization solves the same problem as layer normalization, but it looks in the other direction. It takes one feature, a column, and normalizes it across all the examples in the batch. So after it, each feature has an average of 0 and a spread of 1 over the batch. The price is that an example's result now depends on which other examples happen to be in the batch with it.
About this lesson
Words to know
- batch
- A group of examples that the model processes together in one step.
- normalization
- Rescaling a group of numbers to a standard size, which keeps training stable.
- variance
- A measure of spread: the average of the squared distances from the mean.
Why it matters
Batch normalization is common in image models. Transformers prefer layer normalization, because it gives the same result for an example whatever else is in the batch.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.