What Attention Actually Does

03The turn: where welded wiring gets stuck

Look back at the input line:

x = onehot(a) ++ onehot(b)

This line presupposes that a and b have already been extracted and placed into fixed slots.

But real input is a sequence:

7   *   8   =

Which one is a, which is b, at which position? Order varies, length varies, other tokens can sit in between (a token is the smallest unit the model processes; here it's a single character, vocabulary of 13: 0–9, *, =, ;) — say, multiply 8 by 7. The line can no longer be written.

Part one could write that line because we picked a and b out for the model. Stop doing that, and the convention “slot a holds a” cannot hold — which position to read depends on the content of this particular problem.

token → vector: one-hot collapses again

Each token first becomes a vector of its own — four parallel tracks, initially blind to each other. This step is a lookup in the embedding table (one learned row per token), and it's an old acquaintance:

Train it first — you already know this whole step

Not a line of structure has been explained; why train first? Because the theory of training was finished in part one, and switching models changes not one word of it: the right-hand side below is the same loop — draw a batch, forward, backward, average the slopes, one Adam step; only the inside of fwd/bwd goes from 4 tables to 14 tables, 9,440 numbers in all.

Train it to a perfect score first — everything below dissects this live specimen; every figure reads its numbers straight off it.

Slower than the MLP — a thousand-plus steps: it also has to learn “where to read”. Click it and keep reading while it trains.

The forward pass — single-step those 39 lines of attention

The model stands at the = position, about to output the first character of the answer. All it holds is ='s own 32 dimensions — it must fetch 7 and 8 first. Fetch them from where?

The classic form first — the matrix shape used by the paper and every textbook, eight steps: project Q, K, V; score with Q·Kᵀ; divide by √d_head; set the “future” to −∞ with the causal mask; softmax each row; multiply the weights back onto V. Each line's comment is that step's logic; this code is not an illustration — it is what this page runs, reconciled bit-for-bit against the production implementation at load time:

The single-stepper below walks the production implementation of the same thing — the matrices written out over one block of memory, cell by cell. Classic form for shapes and logic; production form to see where every number comes from:

“standing at the last position” ⇒ the next character is read only from the last position's vector — real source of generate():
   const t = tok.length - 1; … A.prob[t * V + c]        # emitting looks only at row t; no other row takes part

“where to read” ⇒ that position's vector is assembled like this (the move-V source above):
   for (let u = 0; u <= t; u++) s += att[…t*T + u] * v[u*d + …]
   # row t of att ＝ in what proportion each position's v gets mixed into the vector about to emit

“deciding, each time” ⇒ this row of proportions is recomputed on every forward pass (q·k → softmax, just derived) — not looked up

s += q[t*d + …] * k[u*d + …]     # q always carries t — only the emitting row uses its own q; k always carries u — every position's k is queried again and again by all later rows
o += att[t,u] * v[u*d + …]        # v takes no part in scoring; only after the scores are set is it carried off, multiplied by the weight

The routing is computed live — three on-the-spot proofs

Parameters are static; routing is dynamic — Wq and Wk never move after training, yet “who reads whom” differs on every problem. Three proofs, click each:

Proof ① Feed one character, the table grows one row and one column

Same problem, fed one character at a time from the first; after each, take the last row once — as derived above, that row is the proportion in which each position's v mixes into the emitting vector; the table is stamped fresh at every step, not looked up:

Proof ② Swap the multiplicands, the wiring flips wholesale

Same model, same parameters, two problems that differ only by swapping the multiplicands — the pair is chosen by the page itself (all 45 pairs tried; the one whose wiring changes most):

Proof ③ It reads a token it generated itself

The answer to 7*8= has two digits: after emitting 5 it must emit 6. Switch the stepper above to “2nd char” and check the computed weight row for a line connecting to position 4 — the 5 sitting there is a token the model itself generated one step ago:

While we're here, derive the four “why is that there” — the designers' view

The source above walked past four unexplained parts. Each has one forced line of reasoning:

Position vectors (the + Pe inside emb) — because in the attention formula, u is just a summation index:
  scoring sees only the content of k[u], moving sees only the content of v[u]; swap two tokens and the scores travel with them
  ⇒ without position vectors, “7*8” and “8*7” are the same problem in its eyes — “which position” simply doesn't exist as information; it can only be added in

Causal mask (the u ≤ t bound inside att) — because the training target is the next character, and the next character is lying right there in the input:
  bwd source: const tgt = t + 1 < T ? tok[t + 1] : -1   # the answer row t must predict is slot t+1 of the input sequence
  ⇒ allowing a look ahead ＝ allowing copying the answer: loss collapses to 0 and nothing is learned; at generation time the future doesn't exist yet anyway

Residual (the two add(x, …)) — what got carried in is an increment, and must not overwrite the self:
  x still holds the token + position information the FFN and the emitter need later;
  backprop also gets a free through-lane: bwd source dx1[i] += dx2[i]   # slopes pass through the plus sign unchanged — trainable even when deep

LayerNorm (the three ln) — because this page's two most sensitive parts both fear scale:
  the dot product sums dozens of terms, and softmax then raises e to them — drift a little and it saturates (the same account as √d_head)
  ⇒ each layer first pulls every position back to a fixed scale, so scoring stays in the usable range

Multi-head — one head is one reading quota

First separate two words that get blurred together (q, k, v were taken apart at stepper step 3: what I ask with / what I am / what I hand over when chosen):

• Self-attention: q, k, v all come from the same sequence — it looks at itself (as opposed to looking at another sequence; that's cross-attention).
• Multi-head: cut each position's 32 numbers into equal segments, each segment computing its own weights, concatenated back at the end — not copied several times.

Two unrelated adjectives. This model is self-attention + 4 heads.

How exactly “cut into segments” cuts is worth a picture — the crux of attention is this one cut:

The key is softmax: it forces a single head's weights to sum to exactly 1. So one head is one fixed reading quota — give a nine tenths and only one tenth is left for b. And this task needs both operands read clearly. Multi-head ＝ several mutually independent quotas, able to hold several spots at once.

The cost must be stated too: total width is fixed at 32 dimensions — more heads, narrower heads; at 8 heads each gets 32÷8 ＝ 4 dims. It is a trade: more quotas for each seeing more coarsely. Which matters more on this task — the experiment below trains it in front of you.

04FFN: why widen and then shrink back

The operands have been fetched. What happens next was finished entirely in part one: add two vectors, pass a nonlinearity, look up the answer — it is that MLP. The only difference: here the “two vectors” were fetched by the model itself.

One question remains: why is this MLP 32 → 64 → 32?

The answer sits in one sentence from part one

GELU acts dimension by dimension — how dimension 3 is treated depends only on dimension 3's own value.

It cannot express “activate when dimensions 3 and 7 are large at the same time”; it can only judge each dimension alone. Hence a hard constraint:

Negative check: remove the nonlinearity

If this explanation holds, then removing GELU should make those 64 middle dimensions instantly meaningless — provable without training; it is the sentence above.

And in actual training? Same structure, same steps, only no nonlinearity:

What lives in those 64 dimensions

If “one feature, one dimension” holds, it should work in reverse: given a concept, search the 64 dimensions for the one encoding it.

Widening does not buy capacity

Intuition says widening buys “more room to store things”. That claim can be measured directly: stack the 100 problems' activations into a 100×64 matrix and run principal component analysis (splitting the data into mutually independent directions, ranked by how much variation each explains), and count how many directions it takes to hold 90% of the variation.

05Summary: what actually changed between the generations

Reading the two parts together: the model in part one isn't bad — its wiring was welded in advance by a person, and the moment the input's shape changes, that convention collapses. The one step this page adds hands “where to read” back to the model, computed from content. Handling sequences of any length and any order starts from this step.

Nothing here compared accuracy — the 9×9 table has only 100 problems and both models ace it. Its value is being small enough to draw every single wire.