03 / attention
Every word looks back
Before a model writes anything, each word gets to study the words before it and decide which ones matter. That deciding is called attention.
Whose “it” is it?
Read this sentence and notice what your own brain just did: “The robot dropped the ball because it was slippery.” You knew “it” meant the ball. Swap slippery for clumsy and “it” silently switches to the robot. Attention is the machinery that lets a model make that same move.
Click it, then flip the last word between slippery and clumsy. Watch where the lines go.
Questions and name tags
Each word broadcasts two things: a question (“I’m a pronoun, who could I refer to?”) and a name tag (“I’m a physical object”, “I’m an action”). A word compares its question against every earlier name tag; good matches score high. The scores become percentages, and the word blends the matching words together in those proportions. That blend becomes its new, context-aware meaning.
So after attention runs, the word “it” isn’t a bare pronoun anymore. It’s carrying a payload that is mostly ball (or mostly robot, if you flipped the word). Nothing was looked up in a dictionary; the sentence explained itself to itself.
One rule: a word may only look backwards. The model reads the way you write, left to right, so future words are behind a curtain. That’s why the words after your selection go grey in the demo.
Everyone looks at once
You clicked one word at a time, but the model doesn’t take turns. Every word runs its lookback simultaneously, in one big grid: rows ask, columns answer. Darker means “matters more”.
Hover any row: it’s the same picture as the arcs above, one row per word. The empty top-right triangle is the “no peeking ahead” rule.
Many spotlights, not one
One lookback pattern can’t catch everything: “who does it refer to?” is a different question from “which verb does this word belong to?”. So the model runs dozens of these lookbacks in parallel, each with its own questions and name tags. Each one is called a head. In the hero demo, try switching heads: the reference head hunts for pronouns’ owners, while the nearby head just watches recent words, useful for grammar glue.
A real model like the ones behind modern chatbots stacks up to a hundred heads per layer and dozens of layers. But every single one is doing exactly what you just did with your mouse: pick a word, look back, decide what matters, blend.