Report 063 · AI in the Lab
Three separate benchmarks have now put AI agents to work reproducing published research, and all three come back with a headline number in the low twenties. That looks like a converging measurement. It is not. The three tests differ on the one variable that decides everything, which is how much of the original work you hand the agent before you start the clock.
Spend enough years running instruments and you develop a specific reflex about agreement. When three independent methods produce the same number, the honest first question is not "what does this tell us" but "are these three methods measuring the same quantity." Often they are not, and the agreement is a coincidence of scale rather than evidence of anything.
That reflex is the right one to bring to the current crop of AI research-replication benchmarks. PaperBench reports its best agent at 21.0%. CORE-Bench reports its best agent at 21% on the hardest level. ReplicationBench reports its best model at 22%. If you read only headlines you would conclude that AI agents can replicate roughly one fifth of published research, and that three teams have independently confirmed it.
Nobody confirmed anything. The three benchmarks ask three different questions, and one of them hands the agent the finished code.
CORE-Bench came first, from Zachary Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and Arvind Narayanan at Princeton, posted September 2024. It is 270 tasks built on 90 papers across computer science, social science and medicine, and it measures what the authors call computational reproducibility: reproducing the results of a study using the provided code and data. The papers come as CodeOcean capsules, each containing a Readme, a Dockerfile and instructions.
The word "provided" is doing enormous work there, and the benchmark's three difficulty levels are defined by exactly how much gets provided. Quoting the paper's own table:
CORE-Bench-Easy: "Agent is provided the complete code output from a successful run of the code (instead of having to run the code correctly itself)." The agent's job is to read the output and answer questions about it.
CORE-Bench-Medium: "Agent is provided the Dockerfile required to run the code, alongside text-based instructions for running it in a README." Run the Docker command, read the output.
CORE-Bench-Hard: "Agent is provided only the README file with instructions and no Dockerfile." Install the dependencies, work out the right command, run it, read the output.
So the famous 21% is a score on a task that begins with the authors' working code sitting on disk. It is a real and useful measurement of something. That something is closer to "can this agent do a competent DevOps job on an unfamiliar repository" than to "can this agent do science." The spread across levels makes the point better than any argument: CORE-Agent running GPT-4o-mini scored 44.44%, 32.59% and 16.30% on Easy, Medium and Hard. Same agent, same papers, same questions. The only thing that changed was how much of the work was already done, and the score fell by nearly two thirds.
PaperBench, from Giulio Starace and colleagues at OpenAI, posted April 2025, starts somewhere else entirely. Twenty ICML 2024 Spotlight and Oral papers, replicated from scratch, decomposed into 8,316 individually gradable subtasks with rubrics co-developed with the papers' own authors. And critically, verbatim from the paper: "Importantly, we disallow agents from using or viewing paper authors' original codebases (if any)." Their stated reason is precise about what that buys: "This ensures that we are measuring agents' abilities to code and execute complex experiments from scratch rather than the ability to use existing research code."
Read those two sentences next to CORE-Bench's difficulty table and the comparison collapses. PaperBench explicitly forbids the thing CORE-Bench explicitly provides. The two 21% figures are not two readings of one quantity. They are two different experiments that happen to share a numeral.
ReplicationBench, from Christine Ye and colleagues at Stanford and Toronto, posted October 2025, takes a third position: 20 self-contained astrophysics papers decomposed into 111 research-level tasks, each task co-developed with the original paper's authors, graded on final results rather than a human-written rubric. It scores two things separately, and the distinction is the sharpest one in this whole literature: faithfulness, meaning "adherence to original methods," and correctness, meaning "technical accuracy of results." An agent can be correct by accident and unfaithful on purpose, and only a test that separates them will notice.
The astrophysics choice is deliberate and smart. As the authors put it, it is a field where "research relies heavily on archival data and computational study while requiring little real-world experimentation." No wet lab, no instrument time, no reagents. If you want to isolate the reasoning and implementation part of research from the physical part, that is where you go.
There is a second incompatibility underneath the first, and it is the one that would get flagged instantly in a measurement lab.
PaperBench's score is a weighted rubric score: partial credit accumulated across thousands of hierarchical subtasks, so an agent that builds a plausible codebase and gets no result at all still scores well above zero. CORE-Bench's score is task accuracy: you either answered the question about the output correctly or you did not. ReplicationBench grades final numerical results against the paper's own.
A 21% partial-credit score and a 21% pass rate are not the same kind of quantity, any more than a percentage of maximum reading and a percentage of samples passing are the same thing on an instrument report. They share a symbol. That is all they share. This is the same failure mode I wrote about in what "not detected" actually means: a number that looks like a property of the world turns out to be a property of the protocol that produced it.
Here is the part I find genuinely admirable about these papers, and that almost no coverage mentions.
PaperBench cannot be graded by hand at scale. The authors say plainly that "even grading a single replication attempt can take tens of hours for a human expert." So they built an LLM-based judge, and then, because they are careful, they built a separate benchmark to measure the judge. It is called JudgeEval, it compares automated judge outputs against gold labels from human expert judges, and their best judge achieves an F1 score of 0.83 against it.
That is a calibration curve. They characterized their measuring instrument and published its error before they used it to make claims. ReplicationBench does the equivalent, reporting that "score variability can be on the order of 5% between runs" on its smaller extension set, which is a repeatability statement.
Both of those numbers deserve to travel with the headline figures and never do. An agent scoring 21.0% under a judge validated at F1 0.83 is a different claim from an agent scoring 21.0% under a perfect grader, and the difference is not rhetorical. It is the same reason a spectrum needs its noise floor attached, which I got into in what those error bars actually mean.
PaperBench did the thing benchmarks usually skip. They hired eight people currently enrolled in or holding a machine-learning PhD, from Berkeley, Cambridge, Carnegie Mellon, Columbia, Cornell, Purdue, TU Wien and UMass Amherst, screened by CV and a technical test, and gave them the same papers under conditions matched to the agents: the PDF, the addendum, comparable instructions, a single NVIDIA A10 GPU. Three independent attempts per paper.
On a three-paper subset, the best of three human attempts reached 41.4% after 48 hours of effort. On the same subset, o1 scored 26.6%.
Two things follow, and most commentary picks one and drops the other. The first is that the machines are not close to expert humans on this task. The second, which cuts the other way, is that expert humans with a full 48 hours got 41.4%, not 90%. Replicating a modern ML paper from its text alone is brutally hard for the people best equipped to do it. Any read of "AI scores 21%" that treats 100% as the natural target has quietly assumed a ceiling nobody has demonstrated.
And then the detail I keep coming back to, because it describes something real about how these systems fail: "o1 initially outperforms the human baseline during the early stages of the replication attempt, but humans start outperforming the AI agent after 24 hours."
The agent is faster out of the gate and then stops climbing. That is a stamina curve, not a capability ceiling, and it matches ReplicationBench's observation that best-of-N performance runs well above average performance, which the authors read as models being "capable of many replication tasks, but struggle to perform them reliably." Capability and reliability are different axes. A tool that can do the job one time in three is not a tool that does the job.
Now the fact that reframes all of it, and it is buried in CORE-Bench's introduction rather than its results.
Before building the benchmark, the authors surveyed how often published papers are computationally reproducible by humans when the code and data are already available. Their table runs across roughly fifteen fields. In machine learning, one survey found errors in 10 of 28 studies reviewed. Across a multi-field sample of 2,000 studies, 1,480 had computational reproducibility errors. In NLP, 472 of 549. In geosciences, 39 of 41.
And most directly: analyzing the 2022 ML reproducibility challenge, they found that "only 18 of 28 papers that are accompanied by code and data are completely reproducible." In six of those 28 cases, participants could not fully reproduce the results even after corresponding with the original authors.
Sit with that. Human researchers, with the code, with the data, with access to the people who wrote it, fail to reproduce published papers a large fraction of the time. That is the baseline these agents are being scored against.
Which means a replication benchmark is measuring two things at once and cannot separate them: how good the agent is, and how reproducible the corpus is. A low score is consistent with a weak agent. It is equally consistent with a strong agent and a paper whose methods section omits a preprocessing step, a hyperparameter, a random seed, or a dependency version. Nothing in the score tells you which.
This is the structure of the detection-limit problem again. A detection limit is a fact about the laboratory rather than the substance. A replication score is a fact about the corpus at least as much as about the model. Change the papers and the number moves without a single weight changing.
ReplicationBench's authors clearly know this, which is why their dataset is 20 papers described as "reproducible, self-contained" and co-developed with the original authors. They pre-selected for reproducibility. That is the correct methodological move and it has an unavoidable consequence: their score is measured on unusually clean papers, so it should be read as an upper bound on performance against the literature as it actually exists.
Worth flagging, because I do not want to smooth it over. ReplicationBench's abstract says "even the best-performing language models score under 20%." Its results section says "the best-performing model, Claude 4.5 Sonnet, achieves an average score of 22%." Those are in the same version of the same paper.
I have used 22% above and attributed it to the results section, because that is where the measurement lives and the abstract appears not to have been updated with it. It is a small thing. I mention it because this report is partly about not letting a headline number travel unaccompanied, and the cleanest demonstration of that available to me was sitting in one of my own sources.
When you next see a claim that AI can or cannot reproduce scientific research, three questions settle almost all of it.
Was the original code provided? This is the single biggest lever, worth more than the model, the scaffold and the time budget combined. CORE-Bench provides it. PaperBench forbids it. Any comparison that skips this is not a comparison.
Is the score partial credit or pass/fail? A rubric score rewards a good attempt that produced nothing. An accuracy score does not. Both are legitimate; averaging them is not.
What did humans get on the same tasks? If the answer is "nobody measured," the benchmark cannot tell you whether a low score means the agent is weak or the task is hard. PaperBench measured it, and the answer (41.4%) is the most informative single number in this entire literature.
These three benchmarks are good work. I want to be clear about that, because the argument here is not that they are wrong. They are careful, they publish their limitations, two of them characterize their own graders, and all three were built with the cooperation of the authors whose papers are being replicated. That is more methodological hygiene than most fields manage.
The problem is entirely downstream, in what happens to a number once it leaves the paper that defined it. "21% on PaperBench" is a precise statement about twenty ICML papers, from scratch, without the authors' code, graded by a judge with a known F1. Stripped of every one of those qualifiers it becomes "AI can replicate 21% of research," which is not a weaker version of the same claim. It is a different claim, and nobody measured it.
The deeper finding, though, is the one hiding in CORE-Bench's introduction. We built benchmarks to find out whether machines can reproduce our published results, and in the process we documented how often we cannot reproduce them ourselves. Eighteen of twenty-eight, with the code in hand. If an agent and a human both fail to replicate a paper, the interesting question may not be what is wrong with the agent.
U.S. Army combat veteran (Counter-IED / Electronic Warfare), peer-reviewed researcher in microwave spectroscopy, and founder & CEO of Shroombiosis. Consults on laboratory operations, AI, and supplement formulation.