04 / layers
Tiny detectors, stacked deep
Attention lets words talk to each other. Then each word goes off and thinks alone, using millions of tiny yes-or-no machines. Here is one of them.
One neuron, three pipes, one valve
A neuron is a detector for one specific pattern. This one asks: is this word the end of a question? It watches three clues, weighs each through a pipe, pools the evidence in a tank, and only lets a signal out if the total clears the valve. Fat pipes mean strong opinions; the dashed pipe pulls the level down. Drag the sliders and watch the plumbing do the math.
Try maxing out inside a quotation: even a clear question mark can be someone else being quoted, so the neuron shuts the valve.
Where a word thinks alone
Last chapter, attention let every word gather context from its neighbours. What follows is the opposite kind of step: each word retreats into a private workshop and processes what it collected, no peeking at the others. That workshop is called the MLP, and it is nothing more than a wall of detectors like the one above, all reading the same word at once.
Except "wall" undersells it. A single layer of a real model runs tens of thousands of these neurons per word, each with its own hand-of-cards of weights, each hunting a different pattern: one fires on legal jargon, one on the smell of sarcasm, one on "this sentence is about to end". Nobody chose those patterns. They were learned, and most defy tidy labels.
The valve matters more than it looks. Without it, stacking detectors would just be one big averaging machine. Because each neuron can stay silent until its evidence clears the bar, the layer as a whole can make sharp, conditional decisions: this AND that, but NOT while the other thing.
Now stack it
One layer of talk-then-think buys you a little understanding. The trick is repetition: feed the output of one layer into the next, and each round works on the conclusions of the round before. Slide through the stack and watch what one word "knows" pile up.
Scrub the slider both ways: early layers see spelling and grammar, middle layers work out who did what, late layers lean toward what comes next.
The labels are illustrative, in the spirit of what probing research finds when it eavesdrops on real layers; nobody writes them by hand inside an actual model.
Depth is abstraction
Notice what the slider was really showing: each layer's detectors get to fire on the conclusions of the layer below. A layer-one neuron can only spot surface stuff, endings, capital letters, familiar chunks. A layer-six neuron gets "past tense verb" and "the subject was chef" as raw ingredients, so it can afford to detect grander things, like an apology is happening. Height in the stack buys altitude of thought.
That is the entire recipe, by the way. One block of attention (words talk) plus one block of MLP (each word thinks alone), repeated a few dozen times. No hidden extra machinery, no module labelled "understanding". A frontier model is this page's two demos, photocopied deep and trained on most of the internet.
What comes out of the top of the stack is one richly annotated vector per word. Next chapter: how the model turns the final word's vector into an actual guess about what to say.