06 / position
A sense of order
Attention reads every word at the same time, which means it reads nothing in order. Something has to tell the model who came first.
The bag of words problem
Order does more work in a sentence than you might think. Man bites dog is news; dog bites man is a Tuesday. Same three words, opposite story.
Now the catch. The attention mechanism from the last chapters compares words in pairs and adds up the results, and nothing in that recipe ever asks who came first. Left to itself, attention receives your sentence the way you receive Scrabble tiles: as a bag of words. The left panel below is the sentence you build. The right panel is, honestly, everything raw attention would get.
Drag the words into a new order, or focus a word, press Enter to grab it, and steer with the arrow keys. The bag refuses to notice. Then switch on position tags.
Seat numbers
The tags you just switched on are the blunt fix, and it is genuinely what the first transformer models did: before attention gets to look, stamp every word with its seat number. Word one wears a 1, word two wears a 2, and suddenly the bag can tell man bites dog from dog bites man, because the tiles themselves are different.
It works. But absolute seat numbers come with an itch. The phrase the cat sat means exactly the same thing whether it starts at word 3 or word 3,000, yet with absolute stamps those are different tiles, and the model has to learn the same pattern over and over at every address. What a sentence actually cares about is not where its words sit in the document, but how far apart they sit from each other.
The rotation trick
Modern models mostly use something slicker, called rotary position embeddings, RoPE to its friends. Forget stamps; think clock hands. Every word carries a small dial, and the hand turns a fixed amount for each seat the word sits from the start. Seat 1 points a little clockwise of seat 0, seat 2 a little more, and so on, forever.
The payoff: when attention compares two words, what it feels is not where either hand points, but the angle between them. And that angle depends only on how far apart the words are.
Pick any two words, then drag the shift slider to shove invisible padding in front of the sentence. Every hand swings, but the mint angle between your pair refuses to budge.
Why distance beats address
That stubborn mint number is the whole point. Because position lives in rotation, shifting a sentence deeper into a document spins every hand by the same amount, and the angles between words survive untouched. The cat sat feels identical to the model at word 3 and at word 3,000, no relearning required. Relationships are stored as distances, not addresses.
One dial speed cannot do it all, though. A slow hand separates neighbors nicely but barely moves across a whole chapter; a fast hand tracks long spans but blurs words that sit side by side. So real models give every word not one dial but dozens, from lazy to frantic, all turning at once. Flip on more dials above to see a second, faster hand doing its rounds. That spread of speeds is part of why modern models can keep their bearings across very long documents: somewhere in the stack there is always a dial turning at the right pace for the distance that matters.