Build a GPT forward pass
Now the pieces go together. A GPT takes a row of token IDs and returns, for every position, one score for each token in the vocabulary: its guess at what comes next. Inside, three things happen. Each ID becomes a vector that says which token it is and where it stands. Transformer blocks let each position gather information from the positions before it. A final linear layer turns each position's vector into vocabulary scores.
About this lesson
Words to know
- embedding
- A list of numbers that stands for a token. The model learns these numbers during training.
- positional encoding
- Numbers added to a token's embedding to say where in the sequence the token stands.
- causal mask
- The rule that a token may attend only to itself and earlier tokens, never to later ones.
- logit
- A raw score a model gives to one choice, before the scores are turned into probabilities.
Why it matters
tiny_gpt.py has this forward pass with two changes: its block normalizes before each step, not after (norm_first=True), and one more normalization comes before the last layer. Its self-test runs the same check for looking ahead.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.