Training a neural network sounds like magic until you build the part that does the learning. I followed Andrej Karpathy's lecture, The spelled-out intro to neural networks and backpropagation, and built micrograd: an autograd engine that does for single numbers what PyTorch does for tensors. The code is on GitHub.
What a derivative actually says
Forget the school rules for a moment. The derivative at a point answers one question: if I nudge the input by a tiny h, how much does the output move, and in which direction?
def f(x):
return 3*x**2 - 4*x + 5
h = 0.000001
x = 3.0
(f(x + h) - f(x)) / h # ≈ 14, a steep upward slopeWith several inputs you ask the question once per input. For f = a*b + c, nudging a moves the output by b, nudging b moves it by a, and nudging c moves it by 1. Those two patterns are all of backpropagation for + and *:
- Multiplication: the gradient for one input is the other input.
- Addition: the gradient for each input is
1.
A number that remembers
To compute gradients automatically, every number has to remember how it was made. So we wrap each one in a Value:
| Attribute | Meaning |
|---|---|
data | the number itself |
grad | the derivative of the final output with respect to this value |
_prev | the values this one was computed from |
_op | the operation that produced it |
Every operation, such as +, *, tanh or exp, creates a new Value and stores a small _backward function that knows its own local derivative. Run an expression forward and you've built a computation graph.
The chain rule is the whole trick
Going backwards from the output, each node needs two things: the gradient of the node above it, and the local derivative of the operation between them. Multiply them and you have the gradient for the node below.
If a car travels twice as fast as a bicycle, and the bicycle is four times as fast as a walking man, the car is 2 × 4 = 8 times as fast as the man.
Rates of change multiply. For L = (a*b + c) * f with a=2, b=-3, c=10, f=-2, that gives a.grad = 6 and b.grad = -4. Nudge a and b along their gradients and L rises, exactly as predicted.
Automating the order
Calling _backward() by hand works, but the order matters: a node's gradient must be complete before it passes anything on. A topological sort solves that. Walk the graph depth-first, add each node after all its children, then go through the list in reverse:
def backward(self):
topo, visited = [], set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
self.grad = 1.0
for node in reversed(topo):
node._backward()One detail bit me: _backward uses +=, because a value used twice receives gradient from both uses. That's also why you must zero the gradients before every backward pass during training.
You can also build tanh from smaller pieces, (e^2n - 1) / (e^2n + 1), using exp, subtraction and division. The gradients come out identical. As long as each operation knows its local derivative, the chain rule stitches the rest together.
From neuron to network
A neuron multiplies each input by a weight, sums them, adds a bias, and squashes the result through tanh. Three small classes build on each other:
Neuron(nin):ninweights and one bias.Layer(nin, nout):noutneurons that all see the same inputs.MLP(nin, nouts): layers chained one after another.
MLP(3, [4, 4, 1]) has 16 + 20 + 5 = 41 parameters, and every one is a Value, so backprop reaches all of them.
Training
The loop has four steps:
- Forward pass: predict all four examples.
- Loss: the sum of squared errors, a single
Valuethat sits at the root of the whole graph. - Backward pass: zero the gradients, then call
loss.backward(). - Update: step each parameter against its gradient,
p.data += -0.01 * p.grad.
That last line is gradient descent. After enough iterations the loss falls from about 5.27 to about 0.0004, and the predictions land very close to the targets of 1, -1, -1, 1.
What I took away
- A derivative is a slope, and training is just using slopes to nudge parameters.
- Each operation only needs its local derivative. The chain rule does the global work.
- PyTorch's
autogradis this same idea with tensors and far more operations. My micrograd gradients matched PyTorch's, apart fromfloat32rounding.
The full annotated notebook, with every step above runnable, is in the repo.