Why AI scientists fail, in six patterns

The setup

Dhruv Trehan and Paras Chopra built a pipeline of six AI agent modules, one per stage of a research project: idea generation, hypothesis generation, experiment planning, output evaluation, revision and paper outlining. The modules mostly ran on Gemini 2.5 Pro, chosen for its long context. Claude Code wrote and ran the experiment code on cloud GPUs and turned the outline into a full paper. The goal was maximum autonomy with minimal scaffolding.

They limited themselves to machine learning, where every experiment can run entirely in software, and picked three areas: world models, multi-agent reinforcement learning, and AI safety. Starting from roughly 45 to 50 papers per area, the agents generated up to 15 ideas per area, AI reviewers ranked them, and the humans contacted authors of the source papers for feedback before choosing four ideas to run all the way through.

The scorecard

Three of the four failed. The multi-agent RL idea died during execution. Both world-model ideas died at evaluation, when the results turned out not to mean what the pipeline claimed.

The fourth, an AI-safety study, made it. Its paper, "The Consistency Confound: Why Stronger Alignment Can Break Black-Box Jailbreak Detection," was accepted to Agents4Science 2025, an experimental conference that required an AI system as first author. The conference accepted 48 of 254 valid submissions, and the paper passed a code audit. Its reviews were mostly borderline: three AI reviewers scored it 4, 6 and 4, and the human reviewer gave it a 4. Reviewers credited it as a careful negative result, showing that an existing detection method fails, rather than a new technique.

So the honest headline is not "AI writes accepted paper." It's one borderline acceptance out of four tries, after a human-run funnel that started with well over a hundred papers.

The six failure modes

1. Defaulting to the training data. Told to use specific tools and current versions, the agents kept reverting to whatever was common in their training data. Claude Code repeatedly used an outdated command for the cloud platform despite updated documentation in its instructions, fell back to older research libraries, and ignored a dataset's actual field structure in favor of a standard format. When an error came up, it would often blame the library version and go back to the old one, insisting that was right.

2. Implementation drift. When something got hard, the agents quietly made it easier. In one world-model attempt, a long training run hit time limits; the coding agent treated the slowness as a bug and rewrote the method as a simpler approach that no longer tested the original idea. Elsewhere it ran things in "sample" or "test" mode, and the shortcut broke later stages.

3. Losing the thread. Over long sessions the agents forgot earlier decisions. The coding agent invented its own hyperparameters instead of using the planned ones, misreferenced functions it had written earlier, and, when writing the paper, relied on recent files rather than the original idea, producing a draft the authors say read like a list of experiments with no motivation.

4. Overexcitement. This is the one I'd underline. In the authors' words:

The models consistently reported success despite clear failures and overstated the significance of their research contributions.

Their examples set an agent's announcement that a hypothesis was "successfully validated" beside what evaluation actually found: a dummy reward signal, baselines scoring under 10% of their published values, and incorrect gradient passing. In writing, the agent described work as "the first ever paper" in its area or claimed "seminal contributions" regardless of the result. The authors trace much of this to evaluators reading the agents' own summary reports instead of the raw logs. They also suggest training that rewards agreeableness pushes models toward optimistic readings; that's their hypothesis, not something the paper tests.

5. Missing domain knowledge. Papers show polished final configurations, not the craft knowledge it took to get there, and the agents didn't have it. They picked mismatched baselines, such as a discrete-input baseline for a continuous-control task, and kept testing hypotheses when the baseline was performing 95% below established benchmarks, which makes any comparison meaningless.

6. No scientific taste. The agents didn't notice basic design flaws. One world-model hypothesis was too simple to support any conclusion and was run with a single random seed. A plan assumed offline training for a method that requires online learning. A computationally infeasible approach was recommended as fitting in a 6-hour, single-GPU budget. And degenerate output in the one successful project wasn't caught until a human looked.

What the humans actually did

For a project aimed at autonomy, the human role was substantial and the paper is candid about it: humans curated the starting papers, consulted outside experts, reviewed ideas, terminated most failed experiments by hand rather than through the revision agent, supplied access tokens, sanity-checked and copy-edited the paper, and caught the degenerate output. The authors' conclusion is that full autonomy is still in the future and domain experts will remain in the loop "to nudge, course-correct, and verify."

Their fixes are sensible and familiar to anyone who has run a lab: keep early stages abstract and add technical detail later, verify at every stage against raw data rather than an AI's summary of it, plan for failure with a portfolio of hypotheses, separate writing code from running it, and log everything.

What it doesn't show

This is a case study, not a benchmark. It's four attempts, in one field, with one pipeline and the models available in 2025, written up by the team that built the system, and posted on arXiv without peer review (version 1 is the only version as of today). Models are improving quickly and some of these failures may already be smaller. It doesn't prove AI can't do research.

What it does is name the failure modes precisely enough to check for them. And the overexcitement problem doesn't depend on a particular model generation. Any system whose own reports feed the judgment of whether it succeeded needs an outside check on the raw output.

How I use it

When a headline says an AI system "discovered" or "proved" something, I run this list. How many attempts were there, and how many are reported? Who judged success, and did they look at raw data or the system's own report? What did humans pick, prune or fix along the way? Was the method that produced the result the method that was proposed? Is a key result resting on a single run? I applied the same questions to Kosmos's 79% accuracy figure, and they're why the answer is usually more interesting than the headline.

What would change my read: a version 2 of this preprint, a peer-reviewed version, or a larger multi-team study measuring these failure rates across many attempts and newer models. If one lands, I'll update this post in place.