Maintain cached keys and values
While a GPT writes, each new token attends to every token before it. The keys and values of those earlier tokens do not change from one step to the next, so working them out again every time is wasted effort. A KV cache stores them. Each step computes the key and value of the new token only, adds them to the store, and reuses the rest. Memory is limited, so in this exercise the oldest entries are dropped when the store is full.
About this lesson
Words to know
- KV cache
- The stored keys and values of earlier tokens, kept so they are not recomputed at every step of generation.
- query, key, value
- The three roles in attention: a query is what a token looks for, a key is what it can be matched on, a value is what it passes on.
- context window
- The number of tokens the model reads at one time.
Why it matters
A cache is why a chat model keeps a steady pace as its reply grows. The workshop's GPT recomputes instead: its positions are learned, so when old tokens are dropped every kept token's key and value change.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.