Your AI result won't reproduce, even at temperature zero

I spent years pointing microwaves at molecules. The first thing you learn on any instrument is that a single reading is not a result. You run the blank, you run replicates, you find out how much the number moves when nothing about the sample has changed, and only then do you know what a difference means. The spread is not a nuisance you apologize for in a footnote. It is half the measurement.

Language models are now doing measurement work. They classify documents, annotate datasets, extract values out of papers, score responses, sit inside the methods section of real research. And they get read the way nobody would ever read a spectrometer: once.

The defense offered for this is a single setting. Set the temperature to zero, the argument goes, and the model becomes deterministic, so one reading is all the readings. That belief is wrong, it is wrong for more than one reason, and the size of the error is much larger than most people running these pipelines would guess.

What temperature actually controls

A language model does not emit a word. It emits a score for every word in its vocabulary, and then something has to choose. Temperature is the knob on that choosing step. Divide the scores by a large number and the resulting probabilities flatten out, so unlikely words get picked more often, which is what people mean by creative. Divide by a number approaching zero and the distribution sharpens until the highest-scoring word wins every time.

So at temperature zero the selection step stops rolling dice. That much is true, and it is the entire basis of the belief. The problem is that the selection step was never the only place randomness lives.

Everything upstream of it, the arithmetic that produces the scores in the first place, is a very large pile of floating-point operations running on hardware that is allowed to do them in different orders. Change the order of a floating-point sum and you can change the last bit of the answer. Change the last bit of a score and, once in a while, you change which word was highest. Once you change one word, the model is now writing a different sentence, and the difference compounds from there.

The part that is other people

The usual explanation stops at "floating point is not associative and GPUs are concurrent," which is true but incomplete, and the incompleteness matters because it hides something stranger.

Horace He and colleagues at Thinking Machines Lab took the question apart in September 2025 and landed somewhere more specific. The individual GPU kernels used in inference are, in fact, run-to-run deterministic. What they are not is batch invariant. The numerical result for your request depends on how many other requests were being processed alongside it, because the size of the batch changes how the reduction is structured internally, and a different reduction order gives a different last bit.

Their statement of it, verbatim:

the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies!

Sit with what that means. The server batches your prompt together with whatever else arrived in the same window. How busy the service happened to be, at the moment you pressed the button, is part of the computation. It is not part of your prompt, it is not part of your settings, and it is not something you can see or record. As the same write-up puts it, the other users "are not an 'input' to the system but rather a nondeterministic property of the system."

I have never used an instrument with that property. If a spectrometer's reading depended on how many other people were running samples in the building that afternoon, we would not call it noise. We would call it broken, and we would fix it before publishing anything.

They demonstrated it cleanly. One prompt, "Tell me about Richard Feynman," sent one thousand times to Qwen3-235B-A22B-Instruct-2507 at temperature zero. The result was 80 unique completions. All one thousand agreed perfectly through token 102. At token 103 they split: 992 of them said Feynman was born in "Queens, New York" and 8 said "New York City." Everything after that divergence is a different piece of text.

The good news is that this is an engineering problem with an engineering answer. They wrote batch-invariant versions of the relevant kernels, and all 1,000 completions came back identical. The cost was speed. The unoptimized batch-invariant run took 55 seconds against 26 for the standard one, improving to 42 seconds with more work. Determinism is available. It is being traded away for throughput, quietly, by default.

How big is the effect on actual results

Divergent prose is one thing. The question that matters for research is whether it moves the number you report.

A team from Penn State and Comcast AI Technologies ran that experiment directly. Five models (GPT-3.5 Turbo, GPT-4o, Llama-3-8B-Instruct, Llama-3-70B-Instruct and Mixtral-8x7B-Instruct) on eight multiple-choice tasks drawn from the BBH and MMLU benchmarks, in both zero-shot and few-shot settings, for eighty conditions in total. Ten runs each. Their configuration, verbatim: "We set the temperature at 0, top-p at 1, and fix the seed."

Temperature zero. Top-p one. Fixed seed. Same infrastructure, same inputs, same everything.

From the abstract: "We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings."

Those two numbers are different animals and the difference is worth being careful about, because it is the kind of thing that gets flattened in summaries. The 15 percent is the spread between the highest-scoring and lowest-scoring individual runs. The 70 percent is a per-question envelope: the authors mark a question correct if any of the ten runs got it right, and separately mark it incorrect if any run got it wrong, then compare those two totals. The first tells you how much your reported score depends on which run you happened to publish. The second tells you how many questions the model is not actually stable on at all.

To make it concrete, take one cell of their few-shot table. GPT-4o on MMLU college mathematics: best possible accuracy 88.0 percent, median 69.0, worst possible 44.0. That is the same model, same prompts, same settings, ten runs.

The agreement figures for that condition are the ones I keep coming back to. The authors define two. TARa@10 is the fraction of questions where the parsed answer was the same across all ten runs, so "the answer is a)" and "a) is the answer" count as agreeing. TARr@10 is the strict version, where the raw output strings must match character for character. For GPT-4o on college math, TARa@10 was 50.0 percent and TARr@10 was 0.0 percent.

Zero. Out of one hundred questions, there was not a single one where the model returned the same text twice in a row ten times. And on half of them it did not even settle on the same letter.

The strict metric is unforgiving by design, and the authors say so: "TARr@N score is the strictest metric of instability, since any character variation will result in a disagreement." A model can be perfectly correct and score zero on it. That is fine. It is still telling you something real, which is that there is no such thing as "the" output of this system for this input.

Four sources, and only one of them has a knob

A 2026 preprint by Coqueret, Llull, Oswald, Pérignon, Scheuch and Vilhuber, written for economists using models to generate research data, gives the cleanest inventory of where the variation comes from. They list four sources: "deliberate sampling, silent model updates, numerical rounding, or expert routing."

Temperature addresses the first one. That is all it addresses, and they say so plainly: setting temperature to zero "removes deliberate sampling when that option is available, but it does not eliminate the other sources of randomness."

The other three are worth naming because each fails differently. Numerical rounding is the batch-size story above. Expert routing applies to mixture-of-experts architectures, where which sub-network handles your tokens can itself shift. And silent model updates are the one with no technical fix at all: the thing behind a given API name today may not be the thing that was behind it when you ran your experiment, and you will not be told.

Their conclusion for anybody using a commercial API is blunt: "Exact reproduction is therefore generally not possible when using proprietary application programming interfaces." Running open weights locally helps, but only partly, because "reproducibility still depends on the complete hardware and software stack."

And then the sentence that I think should be pinned above every pipeline that pipes a model's output into a statistic:

LLM outputs should be treated as draws from a distribution rather than as fixed measurements.

Why this is a measurement problem, not an AI problem

What I like about that framing is that it takes the whole thing out of the realm of AI opinion and puts it back into ordinary laboratory practice, where it has a hundred years of answers waiting for it.

Nobody in a chemistry lab is scandalized by an instrument with run-to-run variation. Every instrument has it. The scandal would be reporting a single reading as if it were the value, with no replicates and no stated spread. We have a whole vocabulary for this, and I went through the most abused part of it in an earlier report on error bars: the number you put after the plus-or-minus is a claim about what would happen if you did it again.

So the failure here is not that models are unreliable. It is that they arrived looking like software, and software has trained everyone to expect that the same input gives the same output. That expectation is doing damage, because it means the replicates never get run, so the spread is never measured, so it gets reported as zero by omission.

This also puts a floor under a problem I have written about from the other direction. When benchmarks and agent evaluations report a headline percentage, that number was usually produced by one pass. If ten passes of the same setup can span fifteen points, then a two-point gap between two models on a leaderboard is not evidence of anything. I looked at what happens when AI agents are scored on replicating research papers, and at how a benchmark's baseline is really its control condition. Add this on top: without replicates, a benchmark has no error bar, and a ranking without error bars is a ranking of coin flips near the top.

What to do about it

None of this argues against using these tools. I use them. It argues for using them the way you would use any other instrument that drifts.

Run it more than once. If a model output feeds a number in your result, run the whole thing n times and report the spread, not one pass. This costs money and it is the entire fix.

Report the configuration completely. Model identifier including version and date accessed, temperature, top-p, seed, whether it was an API or local weights, and the date range you ran it. A model name with no version is not a method.

Do not present temperature zero as reproducibility. Say what it does, which is remove the sampling step. Say what it leaves alone.

Ask where the classification boundary is. If a model is labeling your data, the items it flips between runs are exactly the ambiguous ones. That set is not noise to be averaged away. It is a map of where your task definition is underspecified, and it is often the most informative output of the whole exercise.

When you read a paper that used one: look for how many runs. If the methods say the model was queried once per item, the paper has one replicate, and you now know how much to trust the third digit.

The signal

Temperature zero is not a determinism switch. It is one control on one of at least four sources of variation, and the largest of the others depends on how busy a data center was when your job ran.

That sounds like an indictment and it is not meant as one. Batch invariance can be engineered; it has been, and the cost is roughly a factor of two in latency, which somebody may eventually decide is worth paying for scientific work. The deeper habit is cheaper than that and available immediately: stop reading the instrument once.

Every real measurement device in a laboratory gives you a slightly different answer each time you press the button. We do not consider that a crisis. We consider it the reason to run replicates. The only genuinely new thing about this instrument is that it produces fluent English, which makes a single reading feel like a conclusion instead of a data point.