Merge a frequent token pair
A GPT does not read letters or whole words. It reads tokens, which are common pieces of text. Byte-pair encoding, or BPE, finds those pieces by itself. It starts with single characters and looks for the two neighbours that appear together most often. It glues that pair into one new token, everywhere, and repeats. Frequent words end up as one token, and rare words are spelled out from smaller pieces.
About this lesson
Words to know
- token
- One unit of text as the model sees it: a character, a piece of a word or a whole word.
- byte-pair encoding
- A way to build tokens: start from characters and keep merging the most frequent neighbouring pair.
- vocabulary
- The full list of tokens a model knows. A token's place in the list gives its ID.
Why it matters
The token IDs a GPT reads and writes come from a tokenizer trained this way. The next two lessons build the simpler character tokenizer that the workshop's GPT uses.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.