Encode token positions
An embedding says which token it is, but not where it stands. Without more help, the model cannot tell in which order the earlier words came: "dog bites man" and "bites dog man" would give the same prediction for the next word. So a second list of numbers is added to each embedding, one that depends only on the position. The classic recipe uses sine and cosine waves. Think of a row of clock hands turning at different speeds: read together, they give every position its own pattern.
About this lesson
Words to know
- positional encoding
- Numbers added to a token's embedding to say where in the sequence the token stands.
- embedding
- A list of numbers that stands for a token. The model learns these numbers during training.
- token
- One unit of text as the model sees it: a character, a piece of a word or a whole word.
Why it matters
The tiny GPT in the workshop learns its position numbers instead, with a second embedding table. Either way they are added to the token embeddings before the first attention layer.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.