01 / tokens
Chopping language into pieces
A model never sees your words. It sees tokens: little chunks of text snapped together from letters. This is the machine that does the chopping.
Watch a sentence become tokens
The tokenizer starts with the dumbest possible reading: one chip per character. Then it consults a rulebook of glue rules, learned in advance by counting which pairs of pieces show up together most often, and applies them greedily: best rule first, until nothing else sticks. Press play.
Type your own sentence and chop it. The ␣ mark is a space: spaces belong to tokens too, which is why “␣straw” and “Straw” are different chips.
Why not whole words? Why not letters?
The obvious plan is a dictionary: one entry per word. It collapses fast. English has hundreds of thousands of words before you count names, slang, hashtags and typos, and the model would go blank the first time you wrote unstoppableness. Every misspelling would be a total stranger.
The opposite plan, one letter at a time, always works and is always terrible. A single e means nothing on its own, and a page of text becomes thousands of tiny steps, so the model spends its whole life spelling instead of thinking.
Tokens are the learned compromise. Before training, an algorithm called byte pair encoding reads a mountain of text and plays one move on repeat: find the two neighbouring pieces that appear together most often, glue them into one, add the glued piece to the vocabulary. Play that move tens of thousands of times and frequency does the design work for you: the earns its own chip, quokka never does.
One practical footnote: this is the unit you pay in. When an AI service bills “per token”, it is counting exactly these chips, so wordy prompts literally cost more.
Why weird splits happen
Because the rulebook was learned from frequency, not from grammar, it has blind spots you can poke. Rare words shatter. Digits stay loose. Other languages get chopped against the grain. Pick a specimen:
This toy knows 67 merge rules; a real tokenizer learns around a hundred thousand, so its chips are bigger. The blind spots are exactly the same kind.
From chips to numbers
One last sleight of hand: once the text is chopped, each token trades its letters for an ID number in the vocabulary. “The” might be token 464; “␣straw” might be 15721. From here on, the model never touches letters again, only these numbers. So how does a bare number come to mean anything? That is the next chapter.