02 / embeddings
Meaning as a map
Before a model can think about a word, it turns the word into a point on a map. Similar meanings land close together. This chapter hands you the map.
Every word gets an address
Each token from the last chapter gets swapped for a list of numbers called an embedding. You can read that list as coordinates: how far east, how far north, and so on. Words stop being spellings and become places.
And once words are places, meaning becomes geometry. Cat sits near dog not because they share letters, but because they turn up in the same sorts of sentences, so they get filed in the same neighbourhood. Distance on the map is disagreement in meaning.
Hover, tap, or tab to any word. Its four nearest neighbours light up, ranked by measured distance.
Direction is meaning
The map has a second trick. It is not just where a word sits, it is which way things point. The little hop from man to woman points the same way as the hop from king to queen. The hop from a country to its capital is another repeatable arrow. Meanings you can name become directions you can reuse.
Which means you can do arithmetic with words: start at king, walk backwards along the man arrow, forwards along the woman arrow, and look around for the nearest word to where you landed.
press run and watch the arrows.
The blue arrow is the measured difference between two words. The dashed copy is that same arrow carried over and re-applied. The landing spot snaps to the nearest real word.
Two dimensions is a polite lie
Full disclosure: we drew this map by hand, in two dimensions, so it fits on your screen and the arithmetic lands cleanly. The distances and arrows are genuinely computed from the coordinates, but the coordinates themselves came from us, not from a model.
A real model writes nothing by hand. It starts with every word at a random spot, reads a trillion words of text, and nudges the points a little every time a prediction goes wrong. Words that behave alike drift together; words that behave differently drift apart. The map is a fossil record of everything the model has read.
And it does not use two numbers per word. It uses thousands. That sounds unpicturable, and it is, but the rules you just played with survive the trip: nearby still means similar, and directions still carry meanings. There are simply enough spare directions that royalty, plural, past tense, and France-ness can each have their own, without elbowing each other.