The Signal Report // AI IN THE LAB
AI in the Lab.
Where machine learning genuinely helps research, and where 'AI discovered X' is doing a lot of the lifting. All reports →
15 in this beat · page 1 of 2
Your AI Result Won't Reproduce
Temperature zero is the setting people reach for when they want a model to behave like an instrument. In a study that set temperature to zero, top-p to one, and fixed the seed, one model returned a byte-identical answer on 0.0 percent of questions across ten runs. The largest source of the variation is not in your prompt or your settings: it is how busy the server was.
Read the report → Report 081 · AI in the Lab · 2026-08-12 · 9 minThe AI Read the Spectrum. Now Count the Guesses.
The most-quoted accuracy figure for AI structure elucidation from NMR is 69.6%. Two things about it: that is the rate at which the right molecule appears somewhere in fifteen guesses, and it was measured on spectra a computer generated. On 106 real spectra, the same framework scored 33.0%. Getting structures out of spectra is my day job, so here is what the metric means.
Read the report → Report 075 · AI in the Lab · 2026-08-09 · 9 minWhere AlphaFold Quietly Breaks the Chemistry
A lab made the wrong protein by accident. Measured, it had twice the helix of the original and came in threes instead of ones. Four AI predictors, given that same sequence, handed back the original to within an angstrom, with charged atoms parked in the greasy core where charge is not allowed to sit. Why the wrong answer looked so right.
Read the report → Report 069 · AI in the Lab · 2026-08-06 · 9 minThe AI That Can't Beat a Straight Line
Five single-cell foundation models, the kind sold as a step toward simulating a living cell, were tested against baselines written to be as stupid as possible. One of them predicts that nothing happens. On the task of finding surprising biology, nothing beat it. The useful part is what a baseline is for, and how easy it is to pick one that cannot win.
Read the report → Report 063 · AI in the Lab · 2026-08-03 · 9 minCan an AI Replicate a Research Paper?
Three benchmarks now test whether AI agents can reproduce published research, and all three report a headline score in the low twenties. That looks like independent confirmation. One of them hands the agent the authors' working code; another explicitly forbids it. Underneath sits a fact about how often humans fail the same test.
Read the report → Report 057 · AI in the Lab · 2026-07-31 · 8 minFake Citations in Real Papers
An audit of 2.5 million biomedical papers found references pointing at studies that do not exist have risen more than twelvefold since 2023. The number is smaller than the headlines suggest, and a citation that does not exist is the one kind a machine can catch. The dangerous version is the citation that resolves.
Read the report → Report 051 · AI in the Lab · 2026-07-28 · 7 minDid the Robot Chemist Discover Anything?
In 2023 an autonomous lab reported making 41 novel compounds in 17 days with no human hands. In January 2026 the paper was corrected: the number is now 36, the word "novel" is gone from the title and abstract, and the closing line no longer says "discovery." A spectroscopist reads the diff, and the measurement step where automation broke.
Read the report → Report 046 · AI in the Lab · 2026-07-25 · 7 minAsking Three AIs Is Not a Second Opinion
Six frontier models from two companies, 50 papers, and an open-ended task where they should have scattered. They converged instead. A 2026 position paper measured it, and the deeper problem is what the published literature never taught any of them: the failures nobody writes down.
Read the report → Report 040 · AI in the Lab · 2026-07-22 · 7 minDid an AI Really Solve an 80-Year Math Problem?
OpenAI made the claim twice. In October 2025 it collapsed in days: the model had found existing papers, not solutions. In May 2026 it held, with a real disproof of an Erdős conjecture and nine mathematicians signing a human-verified note. What changed wasn't the model. It was who checked the work.
Read the report → Report 034 · AI in the Lab · 2026-07-18 · 6 minYour Paper's Reviewer May Be an AI
At ICLR 2026, 21% of the 75,800 reviews were written by a machine. Then a Stanford-led team showed you can raise those scores by having an LLM rewrite your paper, mostly by adding hedges and emphasis rather than better science. A researcher who both submits and reviews on why an AI reviewer is an uncalibrated instrument.
Read the report →