What data leakage does to an AI accuracy claim

Every machine-learning accuracy claim works the same way. You train a model on one pile of data, then grade it on a second pile it hasn't seen. The score on that second pile, the test set, is the number that ends up in the abstract and the press release. The whole thing rests on one assumption: that the test really was unseen, and that it looks like the world the claim is about.

Sayash Kapoor and Arvind Narayanan of Princeton wrote the paper I hand people when that assumption breaks. It was published in Patterns in August 2023, it's open access, and it's still the clearest map of how good-looking results go wrong. Their definition is worth reading slowly:

"Data leakage is a spurious relationship between the independent variables and the target variable that arises as an artifact of the data collection, sampling, or pre-processing strategy."

In plain terms: something in how the data was gathered or prepared lets the model find the answer through a back door. That back door won't exist when the model meets new data, so the reported accuracy is higher than anything you'd see in real use.

How big the problem is

Kapoor and Narayanan didn't audit thousands of papers themselves. They collected 22 existing reviews, each one written by people inside a field who had checked that field's machine-learning papers for errors. Across those reviews, 17 fields had documented leakage, affecting 294 papers in total. The reviews they draw on range from psychiatric neuroimaging and clinical prediction models to genomics, software engineering and computer security. In their words, "data leakage is a pitfall in every single case."

They are careful to call 294 a lower bound, not a rate. Fields describe the same mistake with different words, which makes a systematic search hard, and most reviews read only the papers, not the code, so errors buried in code go uncounted.

Eight ways the answer gets in

The paper sorts leakage into eight types under three headings. The first heading is the textbook stuff. The third is where the subtle, field-specific failures live.

1. Training and test data weren't kept apart. The obvious version is no test set at all. The common version is quieter: filling in missing values, balancing classes or choosing which features to use on the whole dataset before splitting it. Each of those steps lets information about the test set shape the model. Duplicate records that land on both sides of the split do the same.

2. The model used a feature it shouldn't have. The paper's example: a study predicting hypertension that included use of blood-pressure medication as an input. A patient already on those drugs is a patient someone already diagnosed, so the model is partly reading the answer off the chart.

3. The test set isn't drawn from the world the claim is about. This one has three forms. Temporal leakage is testing a "predict the future" model on data from before its training data. Non-independence is when the training and test samples aren't really separate, like different images from the same patient in both piles; the model can learn to recognize the patient instead of the disease. And sampling bias is testing on a convenient slice, like the autism-screening study that dropped the borderline cases, which are exactly the hard ones.

That third family is the one I'd watch hardest in lab work. In spectroscopy, my own field, it's routine to collect many spectra from each sample or each donor. If those spectra are shuffled and split at random, near-copies of the same sample end up on both sides, and a classifier can score beautifully by learning which sample a spectrum came from. It's the same trap as the patient images, and the fix is the same: split by sample or by patient, never by measurement. That's my reading from the bench, applying the paper's category; it's also one more reason to ask what a spectral signal really tracks, as in Report 210.

The case study: civil war prediction

To show what leakage does to conclusions, not just numbers, the authors picked a field where complex models were widely believed to beat the old statistics: predicting civil war onset. A systematic search found 124 papers. Twelve both focused on prediction with a train-test split and shared complete code and data, so those were the ones they could actually check.

They found errors in four of the twelve. In the paper's words, these were "exactly the 4 papers that claimed superior performance of complex ML models over baseline LR models." All four were in top political science journals. The errors were ordinary ones: imputing missing values on training and test data together, reusing an already imputed dataset, and using features that were proxies for the outcome. With the errors fixed, the complex models no longer did substantively better than decades-old logistic regression. In one paper, the gap in AUC (a standard accuracy score) shrank from 0.14 to 0.01.

They found a second, separate problem. Nine of the twelve papers reported no significance test or uncertainty estimate when comparing models. In one, the test set contained only 11 civil war onsets; the reported AUC of 0.85 carried a 95% confidence interval of 0.66 to 0.95, and the proposed model wasn't significantly better than its baselines. A headline number with no error bar on 11 events is barely a measurement.

This is the same pattern as Report 069 and Report 093: once the evaluation is done honestly, the complex model's advantage often shrinks to nothing, or turns out to be the model learning something other than what the paper claims.

When it's medicine

One of the reviews in Kapoor and Narayanan's survey shows the stakes. Roberts and colleagues (Nature Machine Intelligence, 2021) looked at machine-learning models built in 2020 to diagnose or predict COVID-19 from chest X-rays and CT scans. From 2,212 studies identified, 62 passed their quality screen for full review. Their conclusion: "none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases." Kapoor and Narayanan count it among the reviews that flagged leakage, though the abstract's verdict covers a broader set of flaws and biases. Either way, it landed at the moment, as the authors put it, when validated models were urgently needed.

What they propose

The fix Kapoor and Narayanan offer is a "model info sheet": 21 questions a researcher answers alongside the paper, built to force three explicit arguments. First, that the test data never touched training at any step. Second, that every input feature is legitimate for the scientific claim. Third, that the test set comes from the population the claim is about. They're candid about its limits: an info sheet can be filled out wrongly, and without the code and data, nobody can check it.

Questions to ask any "AI predicts X" claim

How was the data split? By patient, by sample, by time, or by random row? Random rows are a red flag whenever one person or sample contributes many records.

Did anything happen before the split? Normalizing, imputing, oversampling or feature selection on the full dataset leaks the test set into training.

Could any input be a stand-in for the answer? A treatment, a test that's only ordered when doctors already suspect the disease, a scanner used only in one clinic.

Was it tested where the claim applies? A different hospital, a later year, a new batch of samples. Same-source test data tells you about the source.

Is there an error bar, and a simple baseline? If a deep model beats nothing, or beats a straw man, or wins by a margin smaller than its uncertainty, the headline is the leak.