Lesson 1 · Backpropagation
A network does not know which of its weights caused a mistake. Backpropagation works it out with a single pass from the output back to the input, multiplying a few local ratios along the way. Follow it here on the smallest network that shows all of it.
Use Next, or the ← → keys. Every number is computed live in your browser.
Everything in the lesson comes from these nine rules. The panel above shows the ones each step uses.
f′(a) ≈ [ f(a + ε) − f(a) ] / εMove a by a tiny amount ε and see how much f moves: the ratio is the slope of f at that point.f(a) = a², a = 3: (3.01² − 9) / 0.01 = 6.01 ≈ 6∂f/∂a = derivative with everything except a held stillWith several variables, we move one at a time. The symbol ∂ says that the others count as plain numbers.f = a·b: ∂f/∂a = b and ∂f/∂b = a(c)′ = 0What does not depend on the variable does not move when the variable moves.∂(7)/∂a = 0(c·a)′ = cIt is a straight line: its slope is the number multiplying the variable.(3·a)′ = 3(f + g)′ = f′ + g′Each term is differentiated on its own and the results are added. The same goes for a difference.(3·a + 7)′ = 3 + 0 = 3(a²)′ = 2·aThe exponent comes down in front, and the power drops by one.at a = 3: 2·3 = 6, as measured in the first rule[ f(g(a)) ]′ = f′(g(a)) · g′(a)If a acts on f only through g, the effects multiply: how much g moves when a moves, times how much f moves when g moves.(3·a + 1)²: 2·(3·a + 1) · 3; at a = 1 it gives 24(f / g)′ = (f′·g − f·g′) / g²The rule for differentiating a ratio of two functions.( a / (a + 1) )′ = (1·(a + 1) − a·1) / (a + 1)² = 1 / (a + 1)²sinh′ = cosh, cosh′ = sinhsinh z = (eᶻ − e⁻ᶻ) / 2 and cosh z = (eᶻ + e⁻ᶻ) / 2. Since (eᶻ)′ = eᶻ and, by the chain rule, (e⁻ᶻ)′ = −e⁻ᶻ, each one becomes the other.sinh′ z = (eᶻ + e⁻ᶻ) / 2 = cosh zNo libraries: this is the whole algorithm, with the same weights you started from above.
import math
x, y = 1.0, 1.0
w1, w2, v1, v2 = 0.6, -0.4, 0.5, -0.3
lr = 0.5
for _ in range(30):
# forward
h1 = math.tanh(w1 * x)
h2 = math.tanh(w2 * x)
y_hat = v1 * h1 + v2 * h2
loss = 0.5 * (y_hat - y) ** 2
# backward: the same path, in reverse
d_yhat = y_hat - y
d_v1, d_v2 = d_yhat * h1, d_yhat * h2
d_h1, d_h2 = d_yhat * v1, d_yhat * v2
d_w1 = d_h1 * (1 - h1 ** 2) * x
d_w2 = d_h2 * (1 - h2 ** 2) * x
# update
w1 -= lr * d_w1
w2 -= lr * d_w2
v1 -= lr * d_v1
v2 -= lr * d_v2
print(loss)To keep the arithmetic readable, this network has no biases and sees one example only. Real networks use batches and tensors, but the rule is exactly this one.
In lesson 2 we add attention to this same network: it is more weights and more multiplications, and the gradient flows through them the same way.