Decode autoregressively
A language model only ever answers one question: what comes next? To write a whole text you ask it again and again. Give it the text so far, take the token it rates highest, add that token to the end, and ask again with the longer text. It stops when it has written enough tokens, or when it produces a special token that means the text is finished.
About this lesson
Words to know
- autoregressive
- Producing output one token at a time, with each new token added to the input for the next step.
- greedy decoding
- Always choosing the token with the highest score.
- end-of-sequence token
- A special token that means the text is finished. It is often written eos.
- bigram model
- The simplest language model: it predicts the next token from the current token alone.
Why it matters
The workshop's GPT uses this loop with two changes. It has no end token, so it always writes a fixed number of tokens. And it draws each token at random, likelier tokens more often, instead of taking the top one.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.