Share keys across query heads
The KV cache holds keys and values for every head of every layer, and for a long text that is a great deal of memory. Grouped-query attention shrinks it. The model keeps all of its query heads but builds fewer key and value heads, and each key/value head is shared by a group of query heads. With 8 query heads and 2 key/value heads, the cache is a quarter of the size.
About this lesson
Words to know
- grouped-query attention
- Attention in which several query heads share one key and value head, to save memory.
- attention head
- One attention calculation. A layer runs several side by side, each with its own queries, keys and values.
- KV cache
- The stored keys and values of earlier tokens, kept so they are not recomputed at every step of generation.
Why it matters
Llama 3 and Mistral 7B use grouped-query attention, with 32 query heads and 8 key/value heads. With equal head counts it is ordinary multi-head attention, and with one key/value head it is called multi-query attention.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.