What Kosmos's 79.4% accuracy measures

The source is "Kosmos: An AI Scientist for Autonomous Discovery" by Ludovico Mitchener and colleagues, posted to arXiv in November 2025. I read version 2 in full, including the supplementary evaluation instructions. It is a technical report with a long author list from the company and academic groups. I couldn't find a peer-reviewed journal version as of this writing, so treat it as a company's own preprint.

What Kosmos is

You give it a dataset and a research objective. It runs repeated cycles of parallel literature searches and data analyses (20 in the headline runs), keeping a structured "world model" so it doesn't lose the plot as it goes. The company says a run reads about 1,500 papers and writes about 42,000 lines of analysis code, then produces reports where each statement cites a paper or a notebook. Launch pricing was $200 a run.

It works on data it's handed, up to about 5 GB, and the authors say it isn't strong on raw images or raw sequencing files. A human curates the input and a human evaluates the output. The authors say plainly that the system is meant to augment scientists, not replace them.

Where 79.4% comes from

The team took three "representative" reports and pulled 102 statements from them. They sorted the statements into three types and asked expert scientists to mark each one supported or refuted, without being shown Kosmos's own code or cited papers. The report gives this breakdown:

• Data analysis statements: 85.5% reproducible
• Literature statements: 82.1% validated against primary sources
• Interpretation statements: 57.9% accurate

The report gives percentages, not counts, but only one set of whole numbers fits all four figures and adds up to 102. By my arithmetic that's 47 of 55, 23 of 28, and 11 of 19, for 81 of 102 overall. That's my reconstruction, not a table from the paper. It matters because the interpretation score rests on just 19 statements. A rough 95% confidence interval on 11 of 19 runs from about 36% to 77%, and on the overall figure from about 71% to 86%. So 79.4% is not a precise property of the system. It's a small sample from three reports the authors picked, and I'd read the one decimal place with that in mind.

Interpretation failed on an easy test

This is the part I think matters most, and it's in the supplement, not the abstract.

Each statement type had its own grading rubric, and the report publishes an example set. Data analysis evaluators were told to try reproducing the claim themselves, and the exact numbers didn't need to match as long as the central idea held. Literature evaluators could mark a claim supported even if parts were ambiguous, as long as nothing was obviously wrong.

The interpretation rubric was the most forgiving of all. Evaluators were told not to run any analysis, to assume the supplied context was correct, and to judge only whether the claim was a logically reasonable inference from that context. So the 57.9% isn't a measure of whether Kosmos's interpretations turned out true. It's a measure of whether they followed from their own premises. Roughly four in ten were marked as failing even that.

The authors' own explanation is that Kosmos tends to confuse statistically significant results with scientifically valuable ones. They also note that it makes excessively strong claims and invents unusual metrics that can be hard to interpret. As a researcher, that rings true. Running a correct test is the easy part. Knowing what the result means, and when it means nothing, is the job. So the system scores best on the most mechanical work and worst on the work closest to what we mean by "scientist."

A few more things the 79.4% doesn't cover. The data rubric checks whether a claim reproduces, not whether the analysis was worth running, and the authors say that directly. A statement marked "unsure" could be clarified and sent back for a definite answer, so ambiguity didn't get its own bin. And accuracy of individual statements doesn't measure whether a report's big conclusion is right. A report can be mostly accurate sentences assembled into a weak story.

What one in five means at report length

The paper says an average report holds 3 to 4 discovery narratives of about 25 claims each. That's 75 to 100 claims. If the 79.4% holds, which is an assumption, a single report would contain something like 15 to 20 wrong statements. They aren't marked. The authors state that no automated method exists to reliably judge whether a claim is accurate, novel and significant, and that finding the valuable discoveries is time-intensive work for domain experts.

The refuted examples in the supplement show what that looks like. In one, an agent mixed up values for some of the mice, and a strong-looking correlation turned out not to be significant when the evaluator redid it. The paper's traceability is a real strength here, because you can open the notebook and check. But it only saves you if someone actually does.

The six-month number

The abstract says a 20-cycle run did the equivalent of 6 months of collaborators' research time. The company's announcement explains how it got that: it polled beta users on how long the same conclusion would have taken them. The average across seven scientists was 6.14 months, with a standard deviation of 2.49. The same post offers a line I wish more AI announcements included: "Human estimates of time saved are intrinsically suspect."

The team backs it with two other estimates. One is a calculation: count the papers and analyses, assume 15 minutes per paper and 2 hours per analysis, and you get about 4.1 expert-months. The other is that three reproduced discoveries took the original humans about 4 months, a figure the company concedes has its own issues. So the range is more like 4 to 6 months of output, by estimate.

None of these count the time to verify the result. That's my point, not theirs, but it's the obvious one, given the section above. In a preLights discussion of the paper, one scientist who tried Edison's free tools reported that its answers matched their own analysis, but that checking new results took long enough that doing the work themselves might have been faster. That's one person's anecdote on the free tools rather than full Kosmos, and the same scientist was impressed that it sometimes reached their conclusions by methods they hadn't considered. Still, it's the right question. We looked at a similar self-report gap in Report 188.

What the paper does show

I don't want to undersell this. The report describes seven discoveries, and three reproduce human findings the system didn't access, including unpublished metabolomics work and a perovskite solar cell result in materials science. (The company flags a caveat on one: that preprint predated the models' training cutoff.) Reproducing real findings from the data alone, across fields, is impressive, whatever the percentage. The authors also make a smart argument: the value isn't that Kosmos is always right, but that wide exploration can surface true and interesting things for a human to vet. That's a fair way to frame a tool.

Where I'd push back is the distance between that framing and the phrase "AI scientist," and the word "autonomous" in the title. The system is stochastic, so separate runs may not land on the same discoveries, and the company says it often runs the same objective several times. It's sensitive to how the objective is phrased. The company said in the launch post that it was still validating discoveries in the wet lab. As for the closing human step, the paper itself calls that crucial.

How I'd read it

If you run a lab, Kosmos looks like a strong junior analyst with unlimited stamina and uneven judgment. Useful, and worth checking line by line before anyone acts on it. Budget review time as part of the cost, because the paper tells you it isn't optional.

And when any AI system quotes an accuracy number, ask three things. Accuracy of what unit? Graded how leniently? And on which slice of the work, because an average can hide the part you care about. Here, "79.4% accurate" and "got the reasoning wrong about four times in ten on a lenient test" describe the same study. I covered a related case in Report 219, and an earlier system from the same group in Report 146.