05 / prediction
Rolling loaded dice
A model never picks the next word. It scores every word it knows, then rolls dice loaded to match those scores. Here, you get to load the dice yourself.
Load the dice yourself
The sentence below has stopped mid-thought, and the model has just scored every candidate for the next word. Twelve of them are shown, already converted into probabilities: the bars are the dice. Temperature reshapes them, top-p trims the weird tail, and the roll button does exactly what the model does: pick one word, at random, in proportion to its bar.
One simplification, so you know we are being honest: a real model re-scores everything after each word it adds. This toy keeps the same scoreboard for every roll, so you can watch the dice instead of chasing them.
Drag temperature down to 0.1 and the dice barely wobble. Push it past 1.5 and keep an eye on soup.
A score for every single word
At every step, the model finishes its thinking with a score for every token in its vocabulary. Not the plausible ones: all of them, fifty thousand or so, including the emoji and the fragments of Finnish. Most score terribly, but nothing scores zero. That full scoreboard is squashed into probabilities, and then something has to choose.
The obvious choice is the top bar every time. That is called greedy decoding, and it turns out to be a personality flaw: greedy models repeat themselves, loop, and sand every sentence down to its most predictable version. A little randomness is not a bug you tolerate. It is what makes the output feel written rather than retrieved.
So you get two knobs. Temperature does not add noise; it reshapes the whole distribution. Below 1, gaps between scores get exaggerated and the favourite becomes a near-certainty. Above 1, the bars flatten out and long shots become live possibilities. Top-p handles the tail: sort the words by probability, keep the smallest set that covers p of the total, and throw the rest away entirely. The greyed-out words in the playground are the ones the dice can no longer land on, no matter how lucky.
Temperature is a personality
Products built on these models rarely ask you for a temperature. They pick one for you, and that choice is most of what people describe as the model’s vibe. Try the three presets: each one animates the knobs into place, then rolls six words in a row so you can see the setting as a personality. Watch precise roll the same word again and again; that is greedy decoding’s whole personality in one line.
Click unhinged twice. You will not get the same line twice. That is what the dial buys.
Why you never get the same answer twice
Ask a chatbot the same question tomorrow and you may get a different answer, and now you know exactly where the fork happens. The scores are the same. The probabilities are the same. But at every single word, dice get rolled, and one early roll that lands on the second-favourite word sends the whole continuation down a different road.
That is the trade the people who tune these systems are making on your behalf: sharpen the dice and you get answers that are reliable but samey; flatten them and you get surprise, plus the occasional confident absurdity. The randomness is a dial, and someone chose where to set it.