Find inactive hidden units
A unit is another word for a neuron. A ReLU unit outputs 0 whenever the value going into it is 0 or below, and when that happens no gradient passes through it. If this is true for every example, the unit contributes nothing and learns nothing: its weights get no signal to change. Such a unit is called dead. Finding how many units are inactive on a batch is a quick health check for a network.
About this lesson
Words to know
- neuron
- The basic unit of a network: a weighted sum of its inputs plus a bias, passed through an activation function.
- preactivation
- The value that goes into an activation function: the neuron's weighted sum plus bias, before the bend.
- ReLU
- The function
max(0, x). It keeps positive numbers and turns everything else into 0. - batch
- A group of examples that the model processes together in one step.
Why it matters
A layer where most units are inactive is wasting its size. A learning rate that is too high and badly scaled starting weights are the usual causes.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.