Build GPT · 4 of 10 · Python · With a year
Batch next-token examples
A language model learns one skill: given some text, guess what comes next. Training examples for that cost nothing to make. Take any stretch of tokens as the input. The target is the same stretch moved one place to the right, so each position's target is the token that follows it. One window of 8 tokens gives 8 guesses to learn from at once.
About this lesson
Words to know
- context window
- The number of tokens the model reads at one time.
- next-token prediction
- The training task of a language model: given the tokens so far, give a high score to the token that comes next.
- batch
- A group of examples that the model processes together in one step.
Why it matters
Every training step of the workshop's GPT starts by cutting a batch of windows like these from the training text, at random start positions.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.