Split before making windows
To know whether a model has learned or only memorised, part of the text is kept aside as validation data and never trained on. With windows there is a trap. Windows overlap, so if you cut all the windows first and then deal them out, a validation window can share most of its tokens with a training window. The model would be tested on text it has nearly seen. The fix is to split the text first and cut windows inside each part.
About this lesson
Words to know
- validation set
- Examples kept out of training and used only to check how the model does on data it has not seen.
- data leakage
- When information from the validation data gets into training, so the model looks better than it is.
- context window
- The number of tokens the model reads at one time.
Why it matters
The validation loss from chapter 4 only means something if the split is clean. The workshop's GPT splits its text this way, so its validation loss is measured on sentences it has not trained on.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.