← The Signal Report Work with me

Report 261 · AI in the Lab

Who graded the AI scientist's papers

A Google team says its autonomous research system, ScientistTwo, takes a published machine-learning paper, beats it, and writes up the result as a paper that outscores accepted human papers. The improvements are measured on real benchmarks. The "outscores" part is a review score, and the reviewers were AI agents. The one that rated it highest is also the one it was trained against, inside its own writing loop.

What the system does

The paper is "ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI," by seven authors at Google Cloud AI Research and the University of Waterloo, posted to arXiv on 17 September 2026. It is a preprint, version 1, not peer reviewed. I read the main text and the benchmark appendix; the remaining appendix pages are sample artifacts I did not review.

The setup is clean. Give the system an accepted paper from ICLR, ICML or NeurIPS, with its code and benchmarks. It finds the method's limitations, proposes ideas, tests them on a slice of the benchmark, runs the survivors at full scale, does ablations, and writes a paper with a runnable codebase. The experiments run on Gemini 3.6 Flash and Claude Opus 4.8, with Claude Code doing the coding.

On 107 such problems, the authors report that ScientistTwo improved on the human method in 86, or 80.4%, with an average relative gain of 25.2% and a median of 7.7%. That gap between mean and median tells you a few big wins pull the average up. It takes about two and a half days and roughly $3,800 per problem.

Those benchmark gains are the strongest part of the paper, because a benchmark number doesn't care who wrote the code. The claim that traveled is different: that the papers are "expert-level, publishable," with "higher average review ratings than human-authored papers." That one depends entirely on who did the reviewing.

Reviewer one is part of the machine

The two graders are AI review agents. The first, ScholarPeer, is a multi-agent reviewer published in January 2026 by a team that shares two authors with ScientistTwo. It is not an outside judge. Inside the pipeline, ScholarPeer reviews each draft on the ICLR 1-to-10 scale, and if the score falls below an acceptance threshold (the paper gives 8 as an example), a Rebuttal Agent runs extra experiments and revises the draft until it clears.

So the system is optimized against ScholarPeer and then scored by ScholarPeer. The authors say so plainly: they call it an "in-distribution evaluation," and they write that it is "not surprising" ScientistTwo scores well on it. On that grader, ScientistTwo's papers average 7.5 with a 91.9% acceptance rate.

To show what that grader thinks of real papers, the authors also ran it on the accepted human papers they started from. ScholarPeer would have rejected 40% of the ICLR 2026 papers in the sample (only five papers), about a third of the NeurIPS 2025 papers, and a fifth of the ICML 2026 spotlights. Papers that passed human peer review, and in the spotlight case were singled out as among the best at the conference, fail this reviewer at a high rate. That doesn't make ScholarPeer useless. It does mean "beats accepted papers on ScholarPeer" is a statement about ScholarPeer.

Reviewer two is the honest test, and it's closer

The second grader is the Stanford Agentic Reviewer, a free tool at paperreview.ai from Andrew Ng's group that was not used during development. The authors call it the held-out evaluator, and it is the number to read.

  • Overall: ScientistTwo's papers average 5.7 with 72.1% predicted acceptance.
  • Against the ICML 2026 spotlights, the largest set (64 human papers, 49 AI papers): the human papers average 6.1 with 96.9% acceptance; ScientistTwo's versions average 5.7 with 69.4%. The human papers win clearly.
  • Against NeurIPS 2025 (38 human, 33 AI): 5.5 versus 5.6, acceptance 76.3% versus 75.8%. A tie.
  • Against ICLR 2026: ScientistTwo comes out ahead, 5.4 versus 5.2, but on four AI papers against five human ones. That is not a sample anyone should generalize from.

One more table is worth a look. After a first round of review and rebuttal, ScholarPeer's acceptance rate for the ICML set was 79.6% and Stanford's was 73.5%. After a second round, ScholarPeer's rose to 93.9% while Stanford's fell to 69.4%. The extra round made the papers better at satisfying the reviewer in the loop and slightly worse in the eyes of the one outside it. That is the textbook signature of optimizing to the grader, and it's the authors' own data.

What an AI review score is, in the first place

Stanford's own description of its reviewer is useful here. When tuned to output a 1-to-10 score by mimicking public ICLR 2025 reviews, its scores correlated with a human reviewer's at a Spearman 0.42. Two human reviewers correlated with each other at 0.41. That is the basis for calling it near human-level, and it is a fair claim. It is also a reminder of how noisy single reviews are: a correlation around 0.4 is weak agreement, for humans and machines alike. The tool also reads only the first 15 pages of a PDF.

I've covered the other side of this before. A Stanford-led study showed that AI review scores can be raised by rewriting a paper without changing the science, and the Sakana paper in Nature showed what "passed peer review" meant once you read the conditions. ScientistTwo is a different case, since its pipeline runs real experiments. But a review score from an AI agent measures how a paper reads to that agent. It is not acceptance.

The parts that hold up

To be fair to the paper, several things in it are better than the headline:

  • Humans did look. Nine experienced reviewers rated 33 of the NeurIPS-derived papers. They endorsed them standalone (3.7 out of 5 overall) and, compared side by side with the human originals, scored them at parity overall (3.0, where 3.0 means no preference). Humans kept a slight edge on methodological rigor (2.9). That is a real, encouraging result, and it is more modest than "outperforms humans."
  • The citation check works. Without a reference-verification step, the system's drafts contained 19 hallucinated references out of about 1,840. With it, zero of 1,814. It also caught and discarded one solution that gamed the task rules (reward hacking).
  • The authors name a ceiling. In their limitations they write that it "does not yet consistently achieve the caliber of spotlight or oral presentations," which matches the Stanford numbers above.

Two caveats the headline drops. The review averages cover only the 86 problems where ScientistTwo succeeded; the 21 failures produced no paper and no score. And the problems were drawn from an existing benchmark set (AutoSOTA's) where each paper comes with runnable code and a measurable target. That is the setting where this kind of system should do best. A detail for readers who open the case-study paper on page 3: its running header reads "Under review as a conference paper at ICLR 2025," a template leftover in a paper that cites 2026 work. A human area chair would have flagged it in the first minute.

The question to ask

When a lab says its AI writes papers that beat human ones, ask three things. Who graded them, and did the system train against that grader? How did the grader score real accepted papers? And what do people who read the papers say? Here the answers are: AI agents, yes for one of them, harshly, and "about as good, slightly less rigorous." That is a meaningful step for automated ML research. It is not a pioneer pushing past human science. If a peer-reviewed version or a fuller human evaluation changes the numbers, I'll update this post in place.

Sources

  1. Jaehyun Nam, Jinsung Yoon, Yanzhou Pan, Yubo Wang, Rui Meng, Parthasarathy Ranganathan and Tomas Pfister, "ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI," arXiv:2609.19644v1 [cs.AI], 17 September 2026. (Primary source. Version 1, 71 pages; main text (pp. 1–18) and benchmark appendix A.1 read from the PDF, later appendix artifacts not reviewed. Source of the pipeline description, models, 86/107 and 25.2% figures, Tables 2 to 10, the in-loop role of ScholarPeer and the example threshold of 8, the in-distribution and held-out labels, the rebuttal-round numbers, the human evaluation, the integrity audit, the cost and time, the case-study header and all quoted phrases. Not peer reviewed.)
  2. Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister and Jinsung Yoon, "ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review," arXiv:2601.22638, 30 January 2026. (Abstract and author list read via the arXiv API, to confirm what ScholarPeer is and the shared authors. Full paper not read.)
  3. Yixing Jiang and Andrew Ng, "Tech Overview," Stanford Agentic Reviewer (paperreview.ai), Stanford ML Group, accessed 22 November 2026. (Source of the 0.41 and 0.42 Spearman correlations, the ICLR 2025 calibration and the workflow; the 15-page limit is from the tool's submission page.)

The rejection shares for ScholarPeer on human papers are 100% minus the acceptance rates in the paper's Table 3 (60.0%, 65.8% and 79.7%); "about a third" and "a fifth" are my rounding. The median gain is from Table 4. Calling the rebuttal-round pattern "optimizing to the grader" is my reading, not the authors'. The "template leftover" note is my inference from the header text; the paper does not comment on it.

Onur Oncer
Onur Oncer

U.S. Army combat veteran (Counter-IED / Electronic Warfare), peer-reviewed researcher in microwave spectroscopy, and founder & CEO of Shroombiosis. Consults on laboratory operations, AI, and supplement formulation.

← All reports