What an AI paper passing peer review measured

The paper is "Towards end-to-end automation of AI research" by Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha and Jeff Clune, from Sakana AI, Oxford, the University of British Columbia and the Vector Institute. It appeared in Nature on 25 March 2026 and is open access. I read the full text. It's careful about its own limits in places the coverage wasn't, so most of what follows comes straight from it.

What the system does

The AI Scientist strings together off-the-shelf language models into a pipeline. It proposes research ideas and checks them against the literature for novelty, writes and debugs experiment code, runs the experiments, makes plots, writes a LaTeX paper with citations, and then reviews that paper with a separate AI reviewer. In the open-ended version described here, the work is split across OpenAI's o3 (ideas and code critique), Anthropic's Claude Sonnet 4 (code), GPT-4o (reading plots) and o4-mini (reviewing).

One scope point matters before anything else: the system only does machine-learning research, where the experiment is a training run on a computer. The authors chose that domain because "experiments typically occur entirely on the computer," and they write that the system "conducts computational experiments only." No wet lab, no instruments, no samples.

The peer-review test, in full

With permission from ICLR's leadership, the workshop organizers and a university ethics board, the team submitted AI-generated papers to a workshop at ICLR 2025, a top machine-learning conference. The workshop, "I Can't Believe It's Not Better" (ICBINB), focuses on negative results: ideas that should have worked and didn't. Reviewers were told some submissions were AI-generated, but not which ones, and could opt out of reviewing them.

Here's the part the headline drops. The system produced many candidate papers, and the researchers "manually filtered the most promising outputs at each stage." They picked three to submit, based on fit with the workshop topic, whether the code actually implemented the idea and ran, and formatting. The paper says nobody edited the selected papers' content. Of those three:

One scored 6, 7 and 6 (an average of 6.33), above the workshop's average acceptance threshold. The organizers said it would very likely have been accepted. It was withdrawn, as planned in advance, because it was AI-generated.
Two did not meet the bar.

Now the context the paper itself supplies. The workshop's acceptance rate was 70%. The ICLR 2025 main conference accepted 32%. The accepted paper reported a negative result at a venue built for negative results. And the team's own human researchers reviewed all three and concluded that none "met the higher bar for a main ICLR conference publication."

The authors' own summary is fair: The AI Scientist "cannot yet meet the standards of top-tier publications nor even do so consistently for workshops." That sentence is in the paper. It didn't make many headlines.

The AI reviewer that grades everything else

Most of the paper's other claims, including the chart showing paper quality rising as newer language models are plugged in, are scored by the team's Automated Reviewer: o4-mini writing five reviews from NeurIPS guidelines, then a meta-review that issues accept or reject. So how good is that reviewer?

The team tested it on real ICLR submissions with known decisions. Its balanced accuracy was 0.69 on older papers and 0.66 on 2025 papers (after the model's training cutoff). Human reviewers, measured in the NeurIPS 2021 consistency experiment where two committees reviewed the same papers, scored 0.66. On that number, the AI matches humans.

Look one column over in the same table, though. The AI reviewer's false positive rate, the share of human-rejected papers it would accept, was 0.45 on older papers and 0.52 on 2025 papers. The human figure was 0.17. Its false negative rate was 0.17 against 0.52 for humans. So it isn't a human-like reviewer with similar accuracy. It's a lenient one: it misses far fewer good papers and lets through roughly half the weak ones. A balanced-accuracy number averages those two errors, which is how a very different instrument can land on the same score.

I spend a lot of my working life on measurement, and this is the first thing I'd check on any detector: not the single accuracy figure, but where the errors fall. A gate that says "pass" to half of what should fail will make everything upstream of it look better than it is. That matters here because the same reviewer produced the finding that the AI's papers "approach borderline acceptability" for workshops and that quality improves with newer models. Those trends may well be real. They're measured with a lenient ruler.

The authors flag one more caveat themselves: the human comparison comes from NeurIPS 2021 papers while the AI was tested on ICLR papers, so "this comparison is not exact."

What goes wrong

The paper lists the common failure modes plainly: naive or underdeveloped ideas, incorrect implementations of the main idea, lack of methodological rigor, experimental errors, duplicated figures, and "many types of hallucinations, such as inaccurate citations."

An independent evaluation of the earlier, publicly released version of the system, by Joeran Beel, Min-Yen Kan and Moritz Baumgart, found a similar picture: 42% of experiments failed from coding errors, novelty checks labeled established ideas as new, and the generated papers cited few and mostly older sources. They also report that each paper cost a few dollars in compute but still needed hours of human involvement. That study tested an older version than the one in Nature, so treat it as context, not a direct rebuttal.

Who wrote it, and who stands to gain

Six of the eight authors are employees of or consultants for Sakana AI, the company that builds the system. The paper discloses this, and the university reviewed the arrangement. That's normal for industry research and doesn't make the results wrong. It's a reason the independent evaluations matter.

How I'd read it

What this paper shows is real and worth taking seriously: a pipeline of current language models can, with humans choosing among its outputs, produce a machine-learning paper good enough to clear a lenient workshop review. The team also ran the experiment responsibly, with consent, blinding and a withdrawal promise made in advance.

What it doesn't show is an AI that does science on its own. Humans filtered every stage. The domain is computer-only experiments. One of three passed the easier of two bars, and none passed the harder one by the team's own judgment. Most of the scaling claims rest on a reviewer that accepts about half of what humans would reject.

When you see "an AI paper passed peer review," ask three things. Which venue, and what's its acceptance rate? How many tries, and who picked the one that went in? And who, or what, is doing the grading in the rest of the paper?