08 / speed
Why it types so fast
A model writes one token at a time, and each token is a full trip through the network. It still outruns your reading speed thanks to three tricks. Here they are, playable.
Stop re-reading your own homework
Since everything comes out one token at a time, speed is just the work per token. And the naive recipe is embarrassing: to pick the next word, re-read the entire conversation from scratch, every single time. The fix is the KV cache: the model keeps notes on every token it has already digested, so each new token only does its own share of the reading.
Play a full sentence with re-read every time, then flip to remember it (cache) and play again. Same sentence, a fifth of the work.
Fuzzier numbers, same brain
Every weight in the model is a number, billions of them, and every single one gets hauled out of memory for every token. The hauling is the bottleneck. So shrink the numbers: store each weight with fewer bits. That is quantization, and it works by snapping every value onto a coarser grid of allowed levels.
Step down the bits and watch the marker snap to the nearest allowed tick. Mostly you can't tell the difference. That's the trick.
A little helper who guesses ahead
One more trick, and it is sneaky: checking is faster than writing. A big model can verify a whole run of proposed tokens in a single pass, the same way you can skim someone else's draft far faster than you could write it. So a tiny, fast model drafts a few tokens ahead, and the big model plays editor. Where the editor agrees, the drafts are kept for free. Where it disagrees, it crosses one out, writes its own word, and the drafter picks up from there.
Grey chips are cheap guesses. One marigold sweep is the big model checking all of them at once. Mint stays, coral gets corrected.
Three trades, one fast chatbot
None of it is free. The cache trades memory for time: keep the notes, skip the re-reading. Quantization trades precision for memory: fuzzier weights, smaller haul. Speculation trades a little extra compute for wall-clock speed: some drafts get thrown away, but you never sit waiting for the big model to write what a small one could have guessed.
All three were running in the chatbot you used this morning. That eerie, faster-than-thought typing is a stack of small bargains.