Why AI rarely predicts "no effect"

The source is "Contemporary AI lacks the imagination to diverge or negate in science" by Honglin Bao, Siyang Wu, Xiao Liu, Sida Li, Shiyun Cao and James A. Evans of the University of Chicago's Data Science Institute and Knowledge Lab. It went up on arXiv in June 2026, and I read version 3 (10 August 2026), which is longer than the first. It's a preprint, so it hasn't been through journal peer review yet.

What they did

The team collected 121,640 full-text preprints posted after 2023, mostly from bioRxiv (68%) and medRxiv (20%), plus chemistry (3%) and social science (9%). They skipped arXiv on purpose, because its papers dominate AI training data. For each paper, an AI model (o3-mini) pulled out the research puzzle and the background, with a filter to strip any sentence too close to the authors' own hypotheses. Then 26 language models, from eight companies, proposed hypotheses from that puzzle alone: 19 standard chat models, 5 reasoning models and 2 "deep research" agents that can search the web.

Each paper's authors got five of those AI hypotheses and rated them on novelty, feasibility, how likely they were to be true, and whether they'd adopt them. In all, 6,749 scientists returned 25,139 rating sets, after the team excluded 22.1% of responses as low quality. That's a large expert sample for this kind of question, and the design is clever: nobody is better placed to judge a follow-up to a paper than the people who wrote it.

What "null hypothesis" means here

This is where the coverage got loose. In statistics, every test has a null hypothesis. That's not what this paper counted. It counted hypotheses that explicitly predict no relationship, no effect or no difference: "A does not change B." The team trained a classifier to spot those statements and report 99.5% cross-validated accuracy, using 2,000 examples labeled by o3-mini and spot-checked by two people.

The abstract says "no model spontaneously proposes null hypotheses, a move humans make more freely." Coverage turned that into "no AI model proposes null hypotheses," which reads like zero. The chart behind it (Figure 2d) shows otherwise. Every model's rate is above zero, and every model is below the human rate, a gap the authors report as significant. Reading off the chart, the highest AI rate came from a small 3-billion-parameter model, Ministral 3B, at about half the human level. The lowest were OpenAI's o3 and o4-mini and the deep-research agent, near zero. The paper also says plainly that humans write these hypotheses "infrequently." So the fair summary is: humans rarely predict no effect, and AI predicts it even more rarely.

That pattern is my favorite detail in the paper, and it's my own reading of the chart, not a claim the authors make: the strongest reasoning models were the least likely to predict no effect. Reasoning bought more diverse ideas (the paper shows that clearly) but not more negative ones.

Why it might happen, and why it matters

The authors' explanation is the file-drawer problem. Science publishes positive findings far more readily than null ones, so the text a model learns from is full of "A is linked to B" and short on "A isn't." They argue that much of what a field knows about what doesn't work lives in failed experiments and lab notebooks that never get written up, so no model can learn it from papers. That's a hypothesis about the cause, not something the survey tests, but it fits what anyone who has worked at a bench knows: the dead ends are rarely in the literature.

Why care? A prediction of no effect is valuable exactly when the field expects one. "This widely assumed link isn't there" is how a lot of bad ideas die. A hypothesis generator that almost never goes there will keep proposing new associations to test and will rarely suggest checking whether an old one holds up.

The finding I'd put first: the AI judges

Plenty of AI-for-science work grades AI ideas with another AI, because expert time is expensive. This study had 25,139 expert ratings to check that shortcut against, and the shortcut mostly failed.

When Gemini 2.5 Flash, DeepSeek R1 and an o4-mini deep-research model rated the same hypotheses with the same rubric, no model's ratings correlated with the scientists' above r = 0.35 on any dimension. They also bunched their scores in the upper middle of the scale, and asking them to use extreme scores barely spread them out. In head-to-head tests (which of two ideas did the scientist prefer?), the standard novelty metrics and top public reward models performed near chance. The best reward model the authors tested scored 53%, 55% and 49% on feasibility, probability and novelty, where 50% is a coin flip.

The team then trained its own model, a Qwen3-14B, on the scientists' preferences. In biology it picked the expert's favorite 69% of the time on novelty, 62% on feasibility and 67% on probability. For a yardstick, they measured how often two human peer reviewers at the same conference agree on which paper is better: 61.0%. So expert taste can be learned to roughly reviewer-level consistency, but only by training on real expert judgments. An off-the-shelf model told to "act like a reviewer" doesn't get there.

If you've followed this beat, that's the thread connecting several reports. When an AI system's output is graded by an AI, ask whether anyone checked the grader against people (see Report 225 on how one AI scientist's accuracy was scored, and Report 034 on AI peer review).

What the scientists themselves preferred

The experts aren't neutral instruments either, and the paper says so. Scientists favored AI ideas that resembled their own work, rating them more feasible and more likely true but less novel. Senior researchers were less willing to adopt AI ideas, though seniority didn't change their quality ratings. And in deciding what to adopt, "likely to be true" outweighed novelty and feasibility by a wide margin (a coefficient of 0.31 against 0.12 and 0.13). AI ideas were rated lowest in the social sciences, which the authors attribute to fields where the core questions are contested.

Where I'd be careful

The paper also analyzes 39 million papers from 2010 to 2025 and reports that after ChatGPT's release in November 2022, the share of abstracts with no null claims rose in both medicine and social science. The timing is striking, but that analysis is observational. Lots of things changed in publishing after 2022, and the paper's language that LLM use "drives" some of these shifts is stronger than a before-and-after comparison can show on its own. The agent-based simulation that follows is, in the authors' words, "stylized," an illustration of a mechanism rather than a measurement of it.

A few other limits. The models generated hypotheses from a puzzle that another AI extracted, which is not how a scientist would actually use one. The null-hypothesis counts depend on a classifier trained on AI-generated labels. Nobody tested any of these hypotheses in a lab. And "spontaneously" is doing real work in the abstract: the study doesn't show what happens if you simply ask a model for no-effect predictions.

How I'd read it

This is a well-designed, large study, and its sober conclusion seems right to me: today's AI is a useful collaborator for brainstorming hypotheses, and a poor substitute for expert judgment about which ones are worth pursuing. If you use these tools in a lab, my practical takeaway (my suggestion, not something the paper tested) is to ask for the negative case explicitly: what result would show this effect isn't real, and which assumed link in this field might not hold? And be skeptical of any AI-scientist claim graded only by AI judges until someone shows those judges agree with people who know the field.