Run an optimizer step
In chapter 1 you worked out the slopes by hand and updated the numbers yourself. PyTorch does both for you. It records every calculation that leads to the loss, and one call walks back through that record and fills in the gradient of every parameter. A second object, the optimizer, then moves the parameters. Your part is to call four things in the right order, every step.
About this lesson
Words to know
- parameter
- Any number in the model that training is allowed to change. Weights and biases are parameters.
- optimizer
- The part of the training code that changes the parameters, using their gradients.
- SGD
- Stochastic gradient descent, the plainest optimizer: each parameter moves downhill by the learning rate times its gradient.
- backpropagation
- Working out the gradient of every weight by going backwards from the loss, one step of the calculation at a time.
Why it matters
Training a GPT is this same loop: clear, loss, backward, step. The workshop's GPT uses another optimizer, AdamW, in place of SGD, and adds one line that limits the size of the gradients.
With a year
How it works, a worked example, the exercise with its tests, three hints and the explained solution are in the app, with a year of Algo150 for $19. The first chapter, Math foundations, is free for everyone: start with Follow a gradient.