Machine learning from scratch: 36 lessons

Math foundations

The few pieces of math every later chapter uses: how a model is scored, and how it is nudged to do better.

• Follow a gradientA model starts with a guess and improves it in small steps.
• Sigmoid and ReLU activationsA neuron first adds up its inputs.
• Turn logits into probabilitiesA model that picks between several choices gives each one a raw score.
• Measure classification lossA classifier is a model that picks one of several choices, called classes.
• Predict with a linear modelThe simplest model is a weighted sum.
• Fit a line with gradient descentNow put two earlier ideas together, the linear model and gradient descent, and let the model learn.

Neural networks

Build a network from single neurons by hand, and see how the error travels backwards through it.

These lessons come with a year of Algo150, $19. What a year includes

• Build a single neuronA neuron is the building block of a neural network, and it does two things you already know.
• Differentiate one neuronA neuron gives an answer, and the answer is off.
• Backpropagate through two layersReal networks have more than one layer, so the error has to travel back through each of them.
• Compose a multilayer perceptronA multilayer perceptron, or MLP, is neurons arranged in layers.
• Scale initial weightsBefore training, every weight needs a starting value.

PyTorch

Do the same math with tensors, the tool real models are written in, and meet three ways to keep numbers at a steady size.

These lessons come with a year of Algo150, $19. What a year includes

• Reshape and reduce tensorsPyTorch stores numbers in tensors.
• Normalize a feature rowAs numbers pass through many layers, they can drift: some rows of values become huge, others tiny, and training…
• Normalize a training batchBatch normalization solves the same problem as layer normalization, but it looks in the other direction.
• Normalize by root mean squareRMS normalization is a lighter version of layer normalization, and many recent language models use it.

Training

Run a real training loop, and learn to read the signs that it is going well or badly.

These lessons come with a year of Algo150, $19. What a year includes

• Run an optimizer stepIn chapter 1 you worked out the slopes by hand and updated the numbers yourself.
• Read training signalsWhile a model trains, you watch two numbers.
• Find inactive hidden unitsA unit is another word for a neuron.
• Build a digit classifier headA classifier for handwritten digits takes an image as a list of numbers and gives ten scores, one for each digit from…

NLP

Turn text into numbers a model can use: tokens, vectors, and a sense of word order.

These lessons come with a year of Algo150, $19. What a year includes

• Look up token embeddingsA model cannot do arithmetic on words, so each token is first given a whole number, its ID.
• Represent text with word countsThe oldest way to turn text into numbers is to count words.
• Classify a sentiment baselineHere is a whole text classifier in a few lines.
• Encode token positionsAn embedding says which token it is, but not where it stands.

Attention & transformers

The step that lets each token look at the tokens before it, and the block that is built around it.

These lessons come with a year of Algo150, $19. What a year includes

• Compute scaled dot-product attentionAttention lets each token collect information from the other tokens, taking more from the ones that matter to it.
• Combine attention headsOne attention calculation gives each token one way of looking at the others.
• Compose residual attention and normalizationA transformer is the same block stacked many times.

Build GPT

Put it together: a tokenizer, training data, a small GPT, and the loop that makes it write.

These lessons come with a year of Algo150, $19. What a year includes

• Merge a frequent token pairA GPT does not read letters or whole words.
• Create a reproducible character vocabularyBefore a model can read text, you must decide which tokens exist and give each a number.
• Preserve Unicode and unknown charactersWith a vocabulary in hand, encoding a text is a lookup: each character is replaced by its ID.
• Batch next-token examplesA language model learns one skill: given some text, guess what comes next.
• Split before making windowsTo know whether a model has learned or only memorised, part of the text is kept aside as validation data and never…
• Build a GPT forward passNow the pieces go together.
• Train next-token logitsTraining a language model uses nothing you have not seen.
• Decode autoregressivelyA language model only ever answers one question: what comes next?
• Maintain cached keys and valuesWhile a GPT writes, each new token attends to every token before it.
• Share keys across query headsThe KV cache holds keys and values for every head of every layer, and for a long text that is a great deal of memory.