Compose residual attention and normalization
A transformer is the same block stacked many times. Inside a block, each token first gathers information from earlier tokens with attention, and then a small network works on each token alone. Two helpers keep a deep stack trainable. A residual connection adds a step's input back to its output, so the step only has to learn a correction. Normalization keeps the numbers at a steady size.
About this lesson
Words to know
- transformer block
- One layer of a transformer: attention, then a small feed-forward network, each with a residual connection and normalization.
- residual connection
- Adding a step's input to its output, so the step only has to learn a change.
- normalization
- Rescaling a group of numbers to a standard size, which keeps training stable.
- causal mask
- The rule that a token may attend only to itself and earlier tokens, never to later ones.
Why it matters
A GPT is token and position embeddings, a stack of these blocks, and one linear layer on top. You assemble exactly that in the next chapter.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.