WebRobot

Lesson 1 · Backpropagation

How a network learns: the error goes back

A network does not know which of its weights caused a mistake. Backpropagation works it out with a single pass from the output back to the input, multiplying a few local ratios along the way. Follow it here on the smallest network that shows all of it.

Use Next, or the ← → keys. Every number is computed live in your browser.

What to take away

The differentiation rules, all together

Everything in the lesson comes from these nine rules. The panel above shows the ones each step uses.

What a derivative isf′(a) ≈ [ f(a + ε) − f(a) ] / εMove a by a tiny amount ε and see how much f moves: the ratio is the slope of f at that point.f(a) = a², a = 3: (3.01² − 9) / 0.01 = 6.01 ≈ 6
Partial derivative: ∂∂f/∂a = derivative with everything except a held stillWith several variables, we move one at a time. The symbol ∂ says that the others count as plain numbers.f = a·b: ∂f/∂a = b and ∂f/∂b = a
Constant(c)′ = 0What does not depend on the variable does not move when the variable moves.∂(7)/∂a = 0
Constant times variable(c·a)′ = cIt is a straight line: its slope is the number multiplying the variable.(3·a)′ = 3
Sum(f + g)′ = f′ + g′Each term is differentiated on its own and the results are added. The same goes for a difference.(3·a + 7)′ = 3 + 0 = 3
Power(a²)′ = 2·aThe exponent comes down in front, and the power drops by one.at a = 3: 2·3 = 6, as measured in the first rule
Chain rule[ f(g(a)) ]′ = f′(g(a)) · g′(a)If a acts on f only through g, the effects multiply: how much g moves when a moves, times how much f moves when g moves.(3·a + 1)²: 2·(3·a + 1) · 3; at a = 1 it gives 24
Quotient(f / g)′ = (f′·g − f·g′) / g²The rule for differentiating a ratio of two functions.( a / (a + 1) )′ = (1·(a + 1) − a·1) / (a + 1)² = 1 / (a + 1)²
sinh and coshsinh′ = cosh, cosh′ = sinhsinh z = (eᶻ − e⁻ᶻ) / 2 and cosh z = (eᶻ + e⁻ᶻ) / 2. Since (eᶻ)′ = eᶻ and, by the chain rule, (e⁻ᶻ)′ = −e⁻ᶻ, each one becomes the other.sinh′ z = (eᶻ + e⁻ᶻ) / 2 = cosh z

The same thing in under 30 lines

No libraries: this is the whole algorithm, with the same weights you started from above.

import math

x, y = 1.0, 1.0
w1, w2, v1, v2 = 0.6, -0.4, 0.5, -0.3
lr = 0.5

for _ in range(30):
    # forward
    h1 = math.tanh(w1 * x)
    h2 = math.tanh(w2 * x)
    y_hat = v1 * h1 + v2 * h2
    loss = 0.5 * (y_hat - y) ** 2

    # backward: the same path, in reverse
    d_yhat = y_hat - y
    d_v1, d_v2 = d_yhat * h1, d_yhat * h2
    d_h1, d_h2 = d_yhat * v1, d_yhat * v2
    d_w1 = d_h1 * (1 - h1 ** 2) * x
    d_w2 = d_h2 * (1 - h2 ** 2) * x

    # update
    w1 -= lr * d_w1
    w2 -= lr * d_w2
    v1 -= lr * d_v1
    v2 -= lr * d_v2

print(loss)

To keep the arithmetic readable, this network has no biases and sees one example only. Real networks use batches and tensors, but the rule is exactly this one.

Next

In lesson 2 we add attention to this same network: it is more weights and more multiplications, and the gradient flows through them the same way.

← The whole path