Look up token embeddings
A model cannot do arithmetic on words, so each token is first given a whole number, its ID. But the ID itself is only a label: token 7 is not seven times token 1. So every ID is swapped for a list of numbers, called an embedding, taken from a table with one row per token. Looking up a token means picking its row. During training the rows change until tokens that behave alike end up with similar rows.
About this lesson
Words to know
- token
- One unit of text as the model sees it: a character, a piece of a word or a whole word.
- token ID
- The whole number that stands for a token.
- embedding
- A list of numbers that stands for a token. The model learns these numbers during training.
Why it matters
The first layer of a GPT is this lookup, done with torch.nn.Embedding. The lesson on positions adds the missing sense of order.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.