Compute scaled dot-product attention
Attention lets each token collect information from the other tokens, taking more from the ones that matter to it. Every token plays three parts. Its query says what it is looking for. Its key says what it has to offer. Its value is what it hands over. A token compares its query with every key, turns the match scores into shares that add up to 1, and takes that share of each value. In a GPT a token may only look at itself and the tokens before it.
About this lesson
Words to know
- attention
- The step in which each token gathers information from other tokens, taking more from those that match what it is looking for.
- query, key, value
- The three roles in attention: a query is what a token looks for, a key is what it can be matched on, a value is what it passes on.
- causal mask
- The rule that a token may attend only to itself and earlier tokens, never to later ones.
- softmax
- The function that turns a list of scores into probabilities: all positive, adding up to 1.
Why it matters
This one calculation is the centre of every transformer. The next two lessons run several copies of it side by side and wrap it into a block.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.