Signal, sampling

Stop Overtuning Temperature

What the knob actually does, why 0 to 1 is the whole map, and why a clear prompt beats fiddling every time.

The Curio Team · 3 min readWritten with AI from the sources below and checked against them before publishing.

Temperature is the knob everyone reaches for and almost nobody needs. Here is what it actually controls. When a model writes, it picks the next token from a probability distribution over thousands of options. Temperature reshapes that distribution before the pick. At zero, the model takes the most likely token nearly every time, so output is close to deterministic. Turn it up, and lower-ranked tokens get a real chance, so the text wanders.¹

The math is simple. Each candidate token has a score, and temperature divides those scores before they become probabilities. A low value sharpens the gap between the front-runner and the rest. A high value flattens it. That is the whole trick. Sampling is not creativity in any deep sense. It is just how much randomness we allow into a mechanical choice.^{1 2}

A clear prompt fixes more than any temperature setting ever will.

Most providers let you go above 1, but the useful band sits between 0 and 1. Below that ceiling, the model stays coherent. Push past it and you invite repetition, drift, and the occasional word salad. For anything with a right answer, extraction, classification, code, structured output, we keep it near zero. For a first draft or a brainstorm, we nudge to somewhere around 0.7. That is the entire map, and we rarely leave it.²

So why does everyone fiddle? Because tuning feels like control, and a bad output feels like a settings problem. It usually is not. When a model rambles or misses the point, the fix is almost never a decimal. The fix is a sharper instruction, a concrete example, or a tighter format spec. We have watched teams burn an afternoon sweeping temperature from 0.3 to 0.9 when a single line, respond in exactly three bullet points, would have solved it in seconds.¹

What about the other sliders? Top-p, top-k, frequency penalty, all of them shape sampling too, and all of them earn far less attention than they get. Top-p caps the candidate pool to the smallest set whose probabilities sum to a threshold. It overlaps heavily with temperature, so changing both at once just muddies your results. Our advice is to move one knob at a time, or better, move none. Pick sensible defaults and change the prompt instead.²

There is a right way to know if a setting matters. Run the same prompt ten times at each value, then read the outputs side by side against your real criteria. Most of the time the differences are cosmetic, and you will feel the pull to tune evaporate. We do this check before shipping any pipeline. It takes ten minutes, and it settles the argument with evidence instead of superstition.

One honest caveat. Temperature zero is not a guarantee of identical output run to run. Floating-point quirks, batching, and model updates all introduce small wobble, so treat zero as low variance, not a hash. If you truly need repeatable results, pin the model version and cache the response. That is a systems decision, not a sampling one.²

Our take is blunt. Set temperature once per task type and forget it. Near zero for correctness, mid-range for variety, and stop there. Then spend the saved energy on the prompt, because that is where the real leverage lives. The knob is a rounding error. Your instructions are the product.