Train next-token logits
Training a language model uses nothing you have not seen. The model gives a row of scores for each input token. Cross-entropy measures how little probability those scores put on the token that came next. Backpropagation and an optimizer step then raise the score of the right token. To keep the moving parts few, the model here is the simplest one possible: a table that predicts the next token from the current token alone.
About this lesson
Words to know
- bigram model
- The simplest language model: it predicts the next token from the current token alone.
- cross-entropy
- The usual loss for choosing between classes: minus the log of the probability the model gave to the right answer.
- logit
- A raw score a model gives to one choice, before the scores are turned into probabilities.
- optimizer
- The part of the training code that changes the parameters, using their gradients.
Why it matters
tiny_gpt.py trains with the same four calls, with the GPT in place of the table. It also limits the size of the gradients and uses the AdamW optimizer, not SGD. A real model should beat a bigram table's loss.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.