Create a reproducible character vocabulary
Before a model can read text, you must decide which tokens exist and give each a number. That list is the vocabulary. The simplest one treats every different character as a token. What matters is that the numbering is the same every time you build it, because a trained model has learned what each number means. Sorting the characters gives that: the same text always produces the same list in the same order.
About this lesson
Words to know
- vocabulary
- The full list of tokens a model knows. A token's place in the list gives its ID.
- token ID
- The whole number that stands for a token.
- token
- One unit of text as the model sees it: a character, a piece of a word or a whole word.
Why it matters
The workshop's GPT builds its vocabulary with this same line and saves it inside the checkpoint. The size of the vocabulary sets the size of the model's first and last layers.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.