Report 069 · AI in the Lab
A team at Heidelberg took five single-cell foundation models, the kind sold as a step toward simulating a living cell, and tested them against baselines written to be as stupid as possible. One baseline predicts that nothing happens. On the genetic-interaction task, nothing beat it. The interesting part is not that the models lost. It is what a baseline is actually for, and how easy it is to pick one that cannot win.
The first thing you learn running a spectrometer is that a peak is not a peak. It is a feature that rose above something, and the something is a blank: the same tube, the same solvent, the same instrument, the same integration time, with the analyte left out. Run the blank, subtract, and only then do you get to have an opinion about the sample.
Nobody publishes the blank. It is boring, it is flat, and it is the entire reason anyone should believe the interesting trace next to it. Take away the blank and a spectrum becomes a picture of your instrument's mood.
Machine learning has the same object under a different name. It is called a baseline, and a benchmark without a serious one is a picture of the benchmark's mood.
Single-cell foundation models are trained on transcriptomes from millions of individual cells, on the theory that a model which has seen enough cells learns something general about how cells work. The ambition has a name now, the "virtual cell": a model you could ask what happens if I knock out this gene, and get an answer without running the experiment. Two of them, scGPT and scFoundation, made the specific claim that they can predict the expression changes caused by genetic perturbations.
That claim is testable, which is the good news, and in August 2025 Nature Methods published the test. Constantin Ahlmann-Eltze, Wolfgang Huber and Simon Anders, at Heidelberg and EMBL, benchmarked seven deep learning models: scGPT, scFoundation, scBERT, Geneformer and UCE on the foundation-model side, plus GEARS and CPA. Their abstract is four sentences long and the third one is the paper:
Here we compared five foundation models and two other deep learning models against deliberately simple baselines for predicting transcriptome changes after single or double perturbations. None outperformed the baselines, which highlights the importance of critical benchmarking in directing and evaluating method development.
"Deliberately simple" is the load-bearing phrase, and it is worth being precise about how simple.
For predicting the effect of perturbing two genes at once, they used two. The first is the "no change" model, which, verbatim, "always predicts the same expression as in the control condition." It has no parameters. It has not been trained on anything. It is a function that returns its input, dressed up as a competitor.
The second is the additive model, which "for each double perturbation, predicts the sum of the individual logarithmic fold changes." If gene A alone does this and gene B alone does that, predict this plus that. It is arithmetic. As the authors point out, neither baseline uses the double-perturbation training data at all.
For the harder task, predicting perturbations the model has never seen, they added a mean baseline, the average across the training perturbations, and a small linear model that represents each gene and each perturbation as a vector and fits one matrix between them.
The results, in the authors' own sentences. On double perturbations: "All models had a prediction error substantially higher than the additive baseline." On the specific task of spotting genetic interactions, where two genes together do something surprising: "None of the models was better than the 'no change' baseline." And on unseen single perturbations: "None of the deep learning models was able to consistently outperform the mean prediction or the linear model."
Sit with the middle one. On the task of finding surprising biology, seven trained models, five of them pretrained on millions of cells at real computational expense, did not beat a function that says nothing happened.
The authors found the mechanism, and it is the same failure I have watched instruments commit for years. Verbatim: "for most genes, the predictions of scGPT, UCE and scBERT did not vary across perturbations," and those of GEARS and scFoundation "varied considerably less than the ground truth."
The models were not wrong in an interesting way. They were flat. Change the input, the output barely moves. In spectroscopy that is a stuck detector, and there is a specific trap that goes with it: a flat output can score well. If most perturbations do very little, then predicting "very little" every time earns a respectable average error while containing no information whatsoever. It looks like accuracy and it is actually a constant.
This is why the "no change" baseline is the right instrument to point at a model like this. It does not measure whether the model is good. It measures whether the model is doing anything. A method that cannot beat a constant has not been shown to have learned the biology; it has been shown to have learned the average.
Then the paper does something cleverer than a knockdown, and this is the part I would put in front of anyone building on these models.
The team pulled the learned gene embeddings out of scGPT and scFoundation, and the perturbation embeddings out of GEARS, and dropped them into their own small linear model, replacing its decoder. The linear model wearing the foundation models' representations "performed as well or better than scGPT and GEARS with their in-built decoders."
So whatever those systems know, a linear map can extract it about as well as the trained network on top of it can. That is a real finding about where the information sits, and it is a more useful one than any leaderboard.
It gets sharper. The linear model's embeddings, when learned from the training data itself, did roughly as well as the ones distilled from millions of cells. Verbatim: pretraining on the single-cell atlas data "provided only a small benefit over random embeddings, but pretraining on perturbation data increased predictive performance." The atlas is not where the signal was. The perturbation experiments are. Which is not a nihilistic conclusion, it is a directive: if you want to predict what happens when you poke a cell, train on data from cells that got poked.
One sentence in the discussion deserves to be quoted carefully, because it is an accusation of a very specific and very ordinary kind, and the authors hedge it. On the comparisons published in the original model papers:
The linear model used in scGPT's benchmark appears to have been set up such that it reverts to predicting no change over the control condition for any unseen perturbation.
"Appears" is theirs and I am keeping it. I have no independent view on that configuration and I am not alleging anyone rigged anything. What I will say is that the structural hazard here is completely general and has nothing to do with machine learning. When the team proposing a method also builds the comparison, the comparison is an artifact of their judgment about what a fair opponent looks like. Every one of us underestimates the opponent we chose ourselves. That is not fraud. It is what a control group exists to prevent, and it is why the control is supposed to be specified by someone with no stake in the outcome.
The Heidelberg team were not proposing a method. They were proposing a blank. It is telling how much that changed.
The obvious defense of any benchmark loss is that the models have moved on. Two 2026 papers speak to that, and I want to flag clearly that both are preprints and neither has been peer reviewed.
The first is VCBench, posted to bioRxiv in June 2026 by Lisa Weidener and colleagues, which evaluates five foundation models including Arc's State model and TranscriptFormer across five testable capability dimensions, against baselines the authors describe as pre-registered. That word matters more than anything else in the paper: the baselines were fixed before the scoring, so nobody got to adjust the opponent after seeing the result. Their finding, verbatim, is that "the baselines match or exceed every foundation model on four of the five scored dimensions, replicating the reported competitiveness of linear baselines on perturbation prediction and extending it to cross-species transfer, GRN inference, and temporal ordering."
Two further things in that preprint are worth more attention than its headline. One model, TranscriptFormer, did clearly win a dimension, cross-modal RNA-to-protein prediction, by 53% on Pearson correlation, and the authors attach their own contamination caveat to it. But the architectural choice that won that dimension, they report, "simultaneously causes a spectral collapse that destroys its temporal-ordering performance, a tradeoff invisible to single-task benchmarks." A model can be genuinely excellent and genuinely broken along axes that no single test will show you at once.
The second is bleaker and is a plain data-hygiene problem: "no foundation model publishes a complete cell-level training manifest, leaving data contamination undetectable to users." If you cannot enumerate what went into training, you cannot prove a test cell was held out. Every score in the field inherits that uncertainty. And of their seven candidate capability dimensions, two, multi-scale integration and in silico experimentation, were judged "structurally untestable as end-to-end tasks" with current architectures and datasets. Those are, not coincidentally, close to what "virtual cell" means in a press release. A claim nobody can currently test is not a claim that is winning. It is a claim that is not yet in the game.
I do not want to leave this looking like the models are worthless, because that is not what any of these papers say, and the Heidelberg authors say the opposite explicitly: "Deep learning is effective in many areas of single-cell omics." Their result is narrow on purpose. It is about perturbation prediction, not about representation learning, annotation, or integration, where these architectures earn their keep.
The best constructive counter-position I found is a February 2026 position paper, also a preprint, from Payam Dibaeinia and colleagues, arguing that the diagnosis is wrong even where the observation is right. Their claim is that "the primary failure mode is a lack of adequate coverage over diverse biological contexts, not insufficient model expressivity," and they explicitly build on the same baseline findings, noting that simple baselines do fine "within a given biological context" while current models "fail to consistently generalize across contexts." They frame it as a causal transport problem, borrowed from causal inference, and conclude that it "cannot be solved by accumulating more data from the same distributions."
I find that persuasive, and note what it does not do: it does not dispute the benchmark. It accepts the loss and argues about the cause. That is how this is supposed to work. Nobody has published a formal rebuttal to the Heidelberg paper that I could find, and the field's response has been more benchmarks rather than complaints about the first one, which is to its considerable credit.
Stated plainly, because they state it plainly. They used only four datasets, chosen because those were the ones the original model papers used. All four are from cancer cell lines, and they note that the Geneformer authors deliberately excluded such lines from training over concerns about mutational burden, which cuts against them. And they did not attempt to clean up the data, including not excluding perturbations that failed to change the expression of their own target gene and so may never have worked.
That last one is a real limitation and it points both ways. Broken perturbations add noise that a "no change" baseline is unusually well suited to exploit. A benchmark on cleaner data might look different. Nobody has run it.
What was the baseline, and who picked it? If a method's own authors specified the comparison, you are reading a claim about their imagination, not about the method. Ask whether anyone with no stake in the answer has run the same test.
Does the output vary with the input? This is the cheapest diagnostic in existence and it catches the most common failure. A model whose predictions barely move when you change the question has learned the average of the training set, and an average can score deceptively well on data where most answers are near the average.
Can the claim be tested at all right now? VCBench's most useful contribution may be ruling two of seven capabilities out of bounds for the moment. Distinguishing "not yet demonstrated" from "not currently demonstrable" is most of what separates a research programme from a marketing document.
The headline finding here is easy to enjoy for the wrong reasons. Big model loses to straight line is a satisfying story, and the satisfaction gets in the way, because the point is not that deep learning failed. The point is that the field ran for several years on comparisons that its own participants designed, and when somebody neutral showed up with a blank, a lot of results changed character at once.
The "no change" model is a null hypothesis in a lab coat. It says: assume nothing happened, and make me prove otherwise. Every measurement discipline that has grown up has grown up by taking that demand more seriously, not less, and usually after an embarrassing period of finding out how many published peaks were the instrument.
This is the same shape as something I wrote about AI replication benchmarks a few days ago, from the other end. There, the problem was a number traveling without the conditions that produced it. Here it is a number traveling without the thing it was measured against. Both are the same missing quantity, and it is the one nobody screenshots.
U.S. Army combat veteran (Counter-IED / Electronic Warfare), peer-reviewed researcher in microwave spectroscopy, and founder & CEO of Shroombiosis. Consults on laboratory operations, AI, and supplement formulation.