Combine attention heads
One attention calculation gives each token one way of looking at the others. But a word relates to its neighbours in several ways at once: one neighbour is its subject, another is the word it describes. So a transformer runs several attention calculations side by side, called heads. Each head has its own queries, keys and values and so its own idea of what matters. Their outputs are then joined end to end into one row per token.
About this lesson
Words to know
- attention head
- One attention calculation. A layer runs several side by side, each with its own queries, keys and values.
- attention
- The step in which each token gathers information from other tokens, taking more from those that match what it is looking for.
- tensor
- PyTorch's block of numbers. It can be one number, a list, a table, or have more dimensions.
Why it matters
The workshop's GPT uses 4 heads, and large models use dozens. The KV cache and grouped-query lessons in chapter 7 are about making all those heads cheaper to run.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.