Preserve Unicode and unknown characters
With a vocabulary in hand, encoding a text is a lookup: each character is replaced by its ID. The interesting part is what to do with a character that is not in the vocabulary, such as an emoji the training text never had. Dropping it would silently change the text. Instead, one ID is kept aside to mean unknown, and every unfamiliar character gets that ID.
About this lesson
Words to know
- token ID
- The whole number that stands for a token.
- vocabulary
- The full list of tokens a model knows. A token's place in the list gives its ID.
- token
- One unit of text as the model sees it: a character, a piece of a word or a whole word.
Why it matters
Every tokenizer needs a rule for text it has not seen. The workshop's GPT takes a stricter path and refuses a prompt with unfamiliar characters; either way the rule is fixed before training and kept afterwards.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.