Know arithmetic? Then you can train a neural network

problem (a,b) → 20-cell input x → 24 cells z → GELU output h → 82 scores u → probabilities p → penalty L
a parameter moves slightly → L changes → gradient → parameter moves back → the next prediction changes

00The task

Input two single digits a and b, output a×b.

The model contains no concept of “multiplication” — only a pile of weights and weighted sums. The times table has to be learned from 100 examples.

The 0–9 table was chosen because it is small enough to draw every single wire: 100 data points, a couple thousand parameters. Every figure below is computed live off the real model — no illustrations.

01Structure and inference

Three layers: 20 input dims on the left — each digit takes 10 slots, only its own slot is 1 and the rest are 0 (this is called one-hot); 24 hidden units in the middle; 82 output dims on the right — 0×0=0 through 9×9=81, every possible answer in the table; whichever scores highest is the answer.

Every wire carries a number, called a weight. A hidden unit's value = each of the 20 inputs times its own wire's weight, summed, plus a bias (one constant per unit). Writing all 24 units together gives z = W1·x + b1; one more such step to the output layer gives u = W2·h + b2.

So this model has exactly four blocks to train: two matrices and two biases. How many numbers each holds, and the total:

What W1 looks like

The nonlinearity in the middle

z cannot be passed on as is; each dimension must first go through an activation function (a nonlinear function applied to a single number). Here it is GELU (Gaussian Error Linear Unit). The formula is one line:

GELU(t) = 0.5 · t · ( 1 + tanh( 0.7978845608 · ( t + 0.044715 · t³ ) ) )

It contains only three things:

• t³ is just t · t · t;
• tanh is an off-the-shelf function whose job is to squash any number into −1 to +1: large numbers approach +1, very negative ones approach −1, and near 0 it passes through roughly unchanged. So the 1 + tanh(…) outside the parentheses lands between 0 and 2, and multiplying by 0.5 makes it a switch between 0 and 1;
• 0.7978845608 and 0.044715 are two hard-coded constants (the first is √(2÷π)), not trained — they are not among the 2,554 parameters counted later.

Read together: GELU(t) ＝ t × a switch between 0 and 1, and the size of that switch is decided by t itself. The larger t is, the closer the switch is to 1 (passes through unchanged); the more negative, the closer to 0 (squashed). Drag the slider:

What W2 looks like

After the nonlinearity we have h; multiply by the second matrix to get scores for the 82 candidate answers:

“Collapsing to two columns” is a special privilege of one-hot input, not a general law of matrix multiplication — h is not one-hot, and not one of its 24 terms can be skipped.

The complete inference code

Chain those steps together and one inference is these six lines. The code on the left can be edited directly, and the right shows the numbers this very line computed — single-step through it, or break something and watch how it fails.

The verdict at the end is wrong: the parameters are still random, so it can only guess. After section 02 trains it, come back here and the same spot will read 56.

Six “why is it built this way” — the designers' view

Section 01 walked past six unexplained decisions. Here is the reasoning for each:

Why one-hot instead of feeding 7 directly? — the model can only multiply and add.
  Feed the raw number and relations like “7 is bigger than 3, adjacent to 8” get forced into every expression,
  yet in the times table the answers to 7×8 and 6×8 are in no sense “adjacent”
  ⇒ one-hot gives each digit its own private column of parameters, mutually unentangled — the digit becomes a lookup key, not a magnitude

Why the middle layer? — first see what happens without the nonlinearity (associativity, two lines):
  u ＝ W2·( W1·x ) ＝ ( W2·W1 )·x        # multiply the two tables into one: two layers collapse into one
  ⇒ without GELU the middle layer adds nothing; the gate has to exist first for the middle layer to exist — what the gate buys is disproved on the spot in part two's FFN section

Why 24 hidden units? — a knob found by trial, same category as the learning rate; no mystical meaning

Why does each unit also get a b? — z ＝ sum of two columns + b: without b, the gate's bend is welded to the origin;
  b is each gate's learnable threshold, moving the “squash zone” to wherever is useful for this task

Why random initialization instead of all zeros? — at all-zero, the 24 units have identical expressions and identical inputs
  ⇒ identical forward outputs ⇒ identical slopes ⇒ still identical after the step ⇒ 24 copies forever, equivalent to one unit
  The sole purpose of the random numbers: to break the tie, giving specialization a chance to grow

Why 82 outputs to “pick one from”, instead of just emitting the number 56? — emit one number and the penalty can only be “how far off”:
  guessing 55 for 56 costs 1, guessing 12 costs 44 — but in a times table being wrong by 1 and by 44 are equally wrong;
  worse, the whole “composed for training” property of softmax + −ln disappears (the core of section 02)
  ⇒ with 82 candidates each holding a slot, “correct” has one definite slot, and the −ln(py) move becomes available

02Training

At the end of the last section the model was wrong: for 7 × 8 it answered 62. This section trains it right — changing only those 2,554 numbers, not one wire.

Pin down one parameter: how 100 problems jointly decide its direction

Choose W1[k,7]: the one weight used by hidden unit k when the first digit is 7. Because x is one-hot, only 7×0…7×9 use it; the other 90 problems give it an exactly zero per-example gradient.

responsibility from problem b:  ∂L7×b/∂W1k,7 = dzk(7,b) × x7 = dzk(7,b)

gradient of the 100-problem mean: ∂L/∂W1k,7 = [ dzk(7,0) + dzk(7,1) + … + dzk(7,9) ] ÷ 100

actual update:                    W1k,7 ← W1k,7 − learning_rate × ∂L/∂W1k,7

①Run it once: get p

Section 01 covered this step; only the last line p = softmax(u) remains. u is the model's score for each of the 82 candidates — it can be positive or negative with no bounds, so it cannot be read as “confidence”. The 82 candidates are numbered (candidate 0 means answer 0, … candidate 81 means 81), so:

c    ＝ any one candidate's index (0…81)　# a placeholder index; “holds for any c” means it holds for all 82
uc   ＝ the score candidate c received
pc   ＝ candidate c's probability　　　pc = e^uc ÷ (e^u₀ + e^u₁ + … + e^u₈₁)
y    ＝ the correct answer's index　　　　# here y ＝ 56, since 7×8＝56
py   ＝ the probability the model gives the correct answer　# here p₅₆ ＝ 0.0041 — it has barely considered it

Each score first becomes e to that power (always positive), then is divided by the total of all 82 (so they sum to exactly 1). 82 non-negative numbers summing to 1 — that is p.

②Measure how far off: loss

For the model to improve itself, we first need one number stating “how bad it is right now”. Using py directly is awkward in two ways: convention wants a penalty where smaller is better; and py dropping 0.50 → 0.49 versus 0.02 → 0.01 both look like “down 0.01”, though the latter is halving the probability. Taking the logarithm and then a negative sign fixes both at once:

penalty for one problem　　L ＝ −ln( py )        # 100% → 0; 10% → 2.30; 1% → 4.61; approaching 0 → to infinity

100 problems fused into one number　L ＝ ( L0×0 + L0×1 + … + L9×9 ) ÷ 100

③Work out which way each parameter should move: the gradient

This is the only difficult step. The whole section does one thing: break “which way should L move” into 2,554 questions of “should this one number go up or down”.

First get clear: L is a function of what?

To differentiate, you must know with respect to what. Trace L's origins backwards:

L　←　the average penalty over 100 problems
　　←　each problem's p　　　　　# computed from W1 b1 W2 b2 and this problem's two digits
　　←　the problem (a, b) and answer a×b　# 7×8＝56 is always 56; immovable
　　←　W1 b1 W2 b2　　　　　# ← only these can move

W1  24 × 20 ＝ 480　　b1  24 ＝ 24　　W2  82 × 24 ＝ 1,968　　b2  82 ＝ 82
                                              ⇒ 2,554 turnable numbers in total

Problems and answers are given constants; x, z, h, u, p are computed intermediates and cannot be set directly. The only knobs on the whole chain are the numbers in those 4 tables. So L is a function that eats 2,554 numbers and emits 1.

Write it out in full and it is section 01's six lines

x = onehot2(a, b)        # the problem spread over 20 numbers: slot a and slot 10+b are 1
z = matvec(W1, x, b1)    # 24 numbers　uses 48 of W1's + all 24 of b1
h = gelu(z)              # 24 numbers　0 parameters
u = matvec(W2, h, b2)    # 82 numbers　uses all 1,968 of W2 + all 82 of b2
p = softmax(u)           # 82 numbers　0 parameters
L = −ln( p[a×b] )       # ← only this line is new

The first five lines are unchanged word for word; line 6 was swapped: there it was answer = argmax(p), picking the highest probability as the answer — that is inference; here we take the probability at the correct answer's slot and compute a penalty — that is training. One and the same forward pass; afterwards one use answers, the other scores.

Substituting py's definition (division becomes subtraction inside ln, and ln(e^uy) is just uy), then averaging the 100 problems, the whole of L written out is these three lines:

L = 1/100 · Σa=0…9 Σb=0…9 [ −ua·b + ln( e^u₀ + e^u₁ + … + e^u₈₁ ) ]

  where   uc = Σk=0…23 W2c,k · GELU( W1k,a + W1k,10+b + b1k ) + b2c

  where   GELU(t) = 0.5·t·( 1 + tanh( 0.7978845608·( t + 0.044715·t³ ) ) )
  Σ just means “add up what follows”: two nested Σ ＝ a and b each run 0–9, one pass per each of the 100 problems

Nudge a little, measure the ratio — that's how a slope is measured

2,554 unknowns is unthinkable, so retreat to a single unknown: that parabola everyone has seen, y = x². At x = 3, nudge x up by 0.01 and y goes from 9.0000 to 9.0601 — the rise ÷ the nudge ＝ 6.01; nudge 0.001 and you get 6.001; nudge 0.0001 and you get 6.0001.

That increasingly steady number 6 is the slope of y = x² at x = 3, also called the derivative. It says exactly one thing: nudge x a little here and y rises 6 times that little.

Back to the 2,554 unknowns. Same method, only hold the other 2,553 still first — with them fixed, L is left as a single-variable curve in this one parameter, and you can nudge and measure the slope just the same.

Ask all 2,554, take one step — and really train it right

Asking once per parameter gives one slope; asking all 2,554 gives 2,554 slopes, lined up in a column that is called the gradient. It points the direction of steepest increase in L, so we go the opposite way:

wi ← wi − learning rate × gi      # gi is this parameter's partial derivative; the learning rate is a knob you set — more in ④

The direction hides in g's own sign: g positive, subtract a positive, it goes down automatically; g negative, subtract a negative, it goes up. One formula, no conditionals. The rest is just repetition: measure all 2,554 partials → everyone takes a small step → measure again. No chain rule, no backpropagation, no matrices needed.

math form                   name in the code (exactly as written in lab.en.js)
────────────────────────────────────────────────────────────
w₁ … w₂₅₅₄　the 2,554 numbers  →  ps = [W1, b1, W2, b2]      # four tables; flattened, that's the 2,554 (line 122)
wi's current value          →  p.d[i]                     # d ＝ data, the number itself
wi's slope gi              →  p.g[i]                     # g ＝ grad, a block the same size as p.d, just for slopes
“run through all 2,554”      →  each((p, i) => …)          # two nested for loops, written once, reused below
L (average penalty over 100) →  averagePenalty()            # called fullLoss() in lab.en.js
learning rate                →  the 5 hard-coded in the code

So the dumb method written out in full is 9 lines:

That sentence, “the causality is inverted”, can be computed in two lines. Every gi is defined as “hold the others still, move only this one”, yet the code moves all 2,554 at once — what stops them from working against each other?

Add up the total change. Each parameter moved by Δwi = −learning rate × gi, and each changes L by gi × Δwi. As long as every step is small enough, the total change is just their sum:

ΔL ≈ g₁·Δw₁ + g₂·Δw₂ + … + g₂₅₅₄·Δw₂₅₅₄　＝　−learning rate × ( g₁² + g₂² + … + g₂₅₅₄² )　＝　−learning rate × Σg²

The dumb method closes the loop here: it really did train it right. And the price is visible too — every single step reruns the full dataset 2,555 times.

Do the accounting first: where the dumb method is slow

The root of the slowness is waste: the intermediates z, h, u computed while measuring parameter 1 are thrown away and recomputed identically while measuring parameter 2 — the same thing torn down and rebuilt 2,554 times. Saving that waste is the entire source of the speedup.

One rule all the way down: the chain rule

Backpropagation has to achieve this: one pass over the data, all 2,554 slopes at once. It relies on a single rule. First see the dependencies clearly — it's one chain:

Follow that chain from right to left:

∂L/∂u  = computed directly              # this is line 1 of the five

∂L/∂W2 = ∂L/∂u × ∂u/∂W2           # upstream × this layer's local derivative
∂L/∂h  = ∂L/∂u × ∂u/∂h

∂L/∂z  = ∂L/∂h × ∂h/∂z            # swap in the new upstream and continue

∂L/∂W1 = ∂L/∂z × ∂z/∂W1
∂L/∂b1 = ∂L/∂z × ∂z/∂b1

Derive the five lines from the definition of the partial derivative

nudge x up by ε and look at the ratio:  ( f(x+ε) − f(x) ) ÷ ε
the derivative ∂f/∂x ＝ the number that ratio settles on as ε shrinks to 0

Apply it to the parabola above: f(x) ＝ x², with letters throughout, no numbers substituted:

  ( (x+ε)² − x² ) ÷ ε ＝ ( x² + 2xε + ε² − x² ) ÷ ε ＝ ( 2xε + ε² ) ÷ ε ＝ 2x + ε
  ε shrinks to 0 ⇒ the derivative of x² ＝ 2x      # a new function holding for every x; put x＝3 and get 6 — exactly the 6.01, 6.001 measured earlier

( (f+g)(w+ε) − (f+g)(w) ) ÷ ε ＝ ( f(w+ε)−f(w) )÷ε ＋ ( g(w+ε)−g(w) )÷ε   # just expand it
  ε shrinks to 0 ⇒ the derivative of a sum ＝ the sum of derivatives; likewise (c·f)' ＝ c × f'

⇒ ∂/∂w [ ( L0×0 + L0×1 + … + L9×9 ) ÷ 100 ] ＝ ( ∂L0×0/∂w + … + ∂L9×9/∂w ) ÷ 100

W2i,j   ＝ in W2, that table of 82 rows × 24 columns, the single number at row i, column j
ui      ＝ the i-th of u's 82 numbers (same for hj, zi, xj)
onehot(y) ＝ a row of 82 slots where only slot y is 1 and the rest are 0     # y ＝ the correct answer's index, set up in ①

Line 1　du = p − onehot(y)　　82 numbers

Three tools first — all of them just counting “how many copies were multiplied”, with one restriction: the base must be positive (roots of negatives get strange; everything here is positive):
  Rule 1　x³·x² ＝ (x·x·x)·(x·x) ＝ x⁵      # same base multiplied ＝ exponents added　⇒ e^a·e^b ＝ e^(a+b)
  Rule 2　(x³)² ＝ (x·x·x)·(x·x·x) ＝ x⁶    # a power of a power ＝ exponents multiplied　⇒ (e^p)ⁿ ＝ e^(np)
  e ≝ ( 1 + 1/n )^n　n＝1 → 2　n＝10 → 2.59374…　n＝1000 → 2.71692…　⇒ e ＝ 2.71828…

ln is the reverse lookup of e^: in the table e^y ＝ w, going from y to w is e^, going from w to y is ln.
  ln e ＝ 1                                # e¹ ＝ e, so looking back gives 1
  The two rules translated to the ln side (write a ＝ e^p, b ＝ e^q, i.e. p ＝ ln a, q ＝ ln b):
  ln(a/b) ＝ ln a − ln b                 # a/b ＝ e^p ÷ e^q ＝ e^(p−q)　(Rule 1)
  ln(xⁿ) ＝ n·ln x                        # xⁿ ＝ (e^p)ⁿ ＝ e^(np)　(Rule 2)

Lemma 1　the derivative of ln w ＝ 1/w　　by the definition, three steps:
  ( ln(w+ε) − ln w ) ÷ ε ＝ ln( (w+ε)/w ) ÷ ε ＝ ln( 1 + ε/w ) ÷ ε      # division becomes subtraction
  write n ＝ w/ε, i.e. ε ＝ w/n (ε shrinking to 0 ⇔ n growing):
    ＝ (n/w)·ln( 1 + 1/n ) ＝ (1/w) · ln( (1+1/n)ⁿ )                 # the coefficient moves into the exponent
  as n grows, (1+1/n)ⁿ closes in on e ⇒ that ln closes in on ln e ＝ 1
                                              # this step borrows “the table is continuous”: nudge the looked-up number a little and the result moves only a little
  ⇒ the derivative of ln w ＝ 1/w
     # check w＝3, ε＝0.001: n＝3000, (1+1/n)ⁿ＝2.71783, its ln＝0.99983, whole ratio＝0.333278 → 1/3 ✓

The other half of the table comes free — same step, swap the axes, and the ratio flips upside down:

Lemma 2　the derivative of e^t ＝ e^t　　# t is any point; later substituted with uc
The same step: t → t+δ　　w ＝ e^t → w+ε          # i.e. t ＝ ln w, t+δ ＝ ln(w+ε); one end shrinking to 0 drags the other with it
  ln side (nudge w, watch t): ( ln(w+ε) − ln w ) ÷ ε ＝ δ ÷ ε　→ 1/w     # Lemma 1
  e^ side (nudge t, watch w): ( e^(t+δ) − e^t ) ÷ δ ＝ ε ÷ δ ＝ 1 ÷ (δ÷ε)   # same pair of numbers, flipped
  ⇒ the derivative of e^t ＝ w ＝ e^t
     # check t＝ln3, δ＝0.001: ε÷δ＝3.001500 → 3 ✓

To prove: ∂L/∂uc ＝ pc − onehot(y)c

L(u) ＝ −uy + ln S, where S ＝ e^u₀ + … + e^u₈₁         # the form derived in “L is a function of what”
By the definition: replace uc with uc+ε (the other 81 unchanged), subtract, divide by ε.

First term　−uy, subtracted:
  c ＝ y: ( −(uy+ε) ) − ( −uy ) ＝ −ε　　　　÷ ε ＝ −1
  c ≠ y: uc does not appear, so the expression is identical before and after ⇒ difference ＝ 0　÷ ε ＝ 0

Second term　ln S — first the change in S; subtract the 82 terms one by one, only e^uc changed:
  ΔS ＝ e^(uc+ε) − e^uc　　ΔS ÷ ε → e^uc          # exactly Lemma 2's ratio with t ＝ uc; as ε shrinks to 0, ΔS shrinks with it
Now the change in ln — split into two ratios multiplied (the chain-rule move):
  ( ln(S+ΔS) − ln S ) ÷ ε ＝ ( ln(S+ΔS) − ln S ) ÷ ΔS　×　ΔS ÷ ε
  ε shrinks to 0: the first settles at 1/S (Lemma 1 with w replaced by S), the second at e^uc
  ⇒ ∂(ln S)/∂uc ＝ (1/S) × e^uc ＝ e^uc/S ＝ pc   # which is exactly softmax's definition

Add the two terms:
  c ＝ y: ∂L/∂uy ＝ py − 1　　　c ≠ y: ∂L/∂uc ＝ pc
  82 of them as a vector: du ＝ p − onehot(y)　82 numbers ✓　QED

Line 2　dW2 = du ⊗ h ， db2 = du　　82×24 + 82

To prove: ∂L/∂W2i,j ＝ dui × hj

First lay out row i of u in full (one row of section 01's matrix multiply: 24 multiplications plus one bias):
  ui ＝ W2i,0·h0 + W2i,1·h1 + … + W2i,j·hj + … + W2i,23·h23 + b2i

Replace W2i,j with W2i,j+ε; h is computed from W1 and x and does not budge.

How much does u change — subtract the 82 rows one by one (the … below are those terms without W2i,j):
  row i: ( … + (W2i,j+ε)·hj + … ) − ( … + W2i,j·hj + … ) ＝ ε·hj   # every other term is identical and cancels
  the other 81 rows: W2i,j does not appear ⇒ difference ＝ 0

How much does L change: in all of u only ui moved, by ε·hj
  Line 1 already proved dui ＝ “per unit ui moves, L moves this much”:
  ΔL ＝ dui × ε·hj

÷ ε, ε shrinks to 0 ⇒ ∂L/∂W2i,j ＝ dui·hj　QED
  Put the 1,968 cells back in the table: row i, column j holds dui·hj — written dW2 ＝ du ⊗ h (outer product)　82×24 ✓
  b2i the same way: Δui ＝ ε ⇒ ΔL ＝ dui·ε ⇒ db2 ＝ du

Line 3　dh = W2ᵀ · du　　24 numbers

To prove: ∂L/∂hj ＝ Σi dui·W2i,j — in matrix form, W2ᵀ·du

Replace hj with hj+ε. How much does u change — subtract the 82 rows one by one:
  u₀ : ( … + W20,j·(hj+ε) + … ) − ( … + W20,j·hj + … ) ＝ W20,j·ε
  u₁ : same way ＝ W21,j·ε
  ⋮                                          # this time every row contains hj — all 82 moved
  u₈₁: same way ＝ W281,j·ε

How much does L change: 82 u's each moved a little at once, contributions add (that rule from the definition):
  ΔL ＝ du₀·W20,j·ε + du₁·W21,j·ε + … + du₈₁·W281,j·ε

÷ ε, ε shrinks to 0 ⇒ ∂L/∂hj ＝ du₀·W20,j + … + du₈₁·W281,j ＝ take W2's column j and du, multiply position by position and add　QED
  The 24 j's each use a column; but matrix multiplication always takes rows ⇒ flip W2 over, rows for columns: dh ＝ W2ᵀ·du　24 ✓
  # the transpose isn't a convention; it is forced by “hj feeds all 82 u's at once”

Line 4　dz = dh ⊙ GELU′(z)　　24 numbers

To prove: ∂L/∂zj ＝ dhj × GELU′(zj)

Replace zj with zj+ε. How much does h change: GELU works dimension by dimension, so only hj moves:
  Δhj ＝ GELU(zj+ε) − GELU(zj) ＝ GELU′(zj)·ε      # GELU′ is defined as exactly this ratio — that curve's slope

How much does L change: only hj moved: ΔL ＝ dhj × GELU′(zj)·ε
÷ ε, ε shrinks to 0 ⇒ ∂L/∂zj ＝ dhj·GELU′(zj)　QED
  All 24 dims mind their own ⇒ dz ＝ dh ⊙ GELU′(z) (⊙ ＝ position-by-position product)　24 ✓
  # when zj is very negative the gate's slope ≈ 0: the forward pass let nothing through, so no upstream slope gets through either

Line 5　dW1 = dz ⊗ x ， db1 = dz　　24×20 + 24

To prove: ∂L/∂W1i,j ＝ dzi × xj — exactly the same derivation as line 2

Replace W1i,j with W1i,j+ε. Subtract z row by row:
  row i: ( … + (W1i,j+ε)·xj + … ) − ( … + W1i,j·xj + … ) ＝ ε·xj
  the other 23 rows: no W1i,j ⇒ difference ＝ 0

How much does L change: only zi moved, by ε·xj ⇒ ΔL ＝ dzi × ε·xj
÷ ε, ε shrinks to 0 ⇒ ∂L/∂W1i,j ＝ dzi·xj　⇒　dW1 ＝ dz ⊗ x　24×20 ✓　QED
  b1 the same way: Δzi ＝ ε ⇒ db1 ＝ dz

Bonus: x is one-hot — 18 of its 20 slots are 0
  xj ＝ 0 ⇒ every cell of that column ＝ dzi·0 ＝ 0 — the whole column's gradient is zero
  ⇒ one problem a×b only changes columns a and 10+b — section 01's “welded wiring”, proved right here

Adding db1 = dz, at this point all 2,554 gradients are complete — one forward pass, one backward pass.

Look back at the bill from “crank it by hand”: the dumb method needs 2,554 reruns of all 100 problems to fill in 2,554 partials; these five lines do the same job in a single pass. What they compute is the same batch of numbers.

Into the loop: the saved waste is one variable

Same starting point and same stepping rule as cranking by hand; only the slopes now come from these five lines. The code gains two names, and they are exactly that discarded intermediate from the accounting section:

as written in the five lines     name in the code
────────────────────────────────────────────────────────
the 5 forward lines              fwd(a, b)　computes, then hands back the intermediates in a bundle
that bundle of intermediates     act ＝ { z, h, u, p }
the 5 backward lines             bwd(act)
dW2 db2 dW1 db1 (the wanted)     W2.g　b2.g　W1.g　b1.g　# no new variables; they land straight in .g
du dh dz (just passing)          temporaries, discarded after use　　　# as the chain-rule section said: carriers, not the goal
correct answer index y           a×b

Both beats end here: the two routes compute the same batch of slopes; the difference is only speed. One of the four things remains — how far to move.

④Take a small step: learning rate and Adam

The gradient says which way, not how far. That “how far” is the learning rate — it appears in both beats' code: hard-coded as 5 in the dumb method's 9 lines, and 5 again in backprop's 13. Too small and it can't move; too large and it overshoots (the “as long as every step is small enough” in beat one's proof is exactly its price). Same model, three learning rates, one training run each:

One learning rate has to serve 2,554 parameters whose gradients differ by orders of magnitude. Adam patches exactly that: accumulate momentum to cancel jitter, then normalize each direction by its own historical magnitude — effectively giving every parameter its own learning rate. At heart it is still a small step down the negative gradient; only the step size is no longer one-size-fits-all.

What gradient descent guarantees, and what it doesn't

ΔL ≤ 0 only says “the next step is lower”; it never says you reach the lowest point. Gradient descent uses only information at the current point, start to finish. So where does it stop? Retrain from a different random start and find out — if there were only one lowest place, every starting point should slide into the same bottom:

Same score, completely different parameters. Combinations that get all 100 right are plentiful, and gradient descent merely found a nearby one. It was never looking for “the optimum”; it looks for “one that will do”.

Not every step needs all 100: the batch

Up to here, every step has computed all 100 problems — L is defined as that average, and nothing was skipped: the dumb method reran 100 problems per parameter measured, and backprop ran all 100 forward and backward on every step.

But that's expensive. And that average can be estimated from a few samples — a bit rougher, with the direction broadly intact. The handful drawn is called a batch, and its size the batch size.

Final version: that loop above, with two changes

Nothing new. Take the loop you stepped through by hand in “into the loop”, bolt on Adam and the batch, and you have the training code everyone actually uses. Compare line by line: only two lines changed (the four things from the opening figure, ①②③④, marked on the right):

for (let step = 1; ; step++) {
    each((p, i) => p.g[i] = 0)               # zero the slopes
    for (const [a, b] of this drawn batch) {      ← change 1: was all 100; now a drawn batch
        const act = fwd(a, b)                # ① run it once
        loss = −ln act.p[a×b]                # ② measure how far off — for your eyes only; training doesn't use it
        bwd(act)                             # ③ the five lines folded into one call; slopes accumulate into p.g
    }
    each((p, i) => p.g[i] /= this batch's size)     # take the average
    adam(ps, 0.02, step)                     ← change 2: was p.d[i] −= 5 × p.g[i]
}

Watch it train

This model runs in your own browser — no server, no pretrained weights. Click “train 1 step” to watch the loss move, then “train until all correct”; it converges in about 100 steps.