The force field that passed the benchmark

A machine-learned force field, or MLFF, is a neural network or kernel model trained to reproduce the energies and forces that quantum chemistry would give you, at a fraction of the cost. It is one of the most genuinely useful applications of machine learning in physical science, and it is not a chatbot doing science: it is a fitted function replacing an expensive calculation inside a simulation you already knew how to run.

Which is exactly why the way these models are graded deserves scrutiny. The number in the abstract is almost always an error on a held-out test set, a mean absolute error in energy or in force, quoted in meV or meV per ångström. Lower is better, papers are compared on it, and it is a real number honestly measured.

It is also not the number that tells you whether your simulation will hold together.

A benchmark that runs the thing

The TEA Challenge 2023 was reported in two companion papers in Chemical Science in February 2025, from a large collaboration including the developers of most of the models tested. The setup is the interesting part.

Five architectures took part: MACE and SO3krates, both equivariant neural networks, and sGDML, SOAP/GAP and FCHL19*, all kernel-based. Four challenge systems: a flexible alanine tetrapeptide, a larger peptide, an organic molecule sitting on a graphene surface, and the perovskite methylammonium lead iodide. Everyone got the same training, validation and test sets, archived publicly. The participants were not involved in preparing the challenge.

Then comes the design decision that makes this worth writing about. The organisers did not accept submitted results. In their words, after a workshop where participants could still improve their models, "the TEA 2023 team performed all the test simulations afterwards using the same HPC hardware and settings for all provided MLFF models to ensure a fair comparison."

That is a benchmark run by the referee rather than by the players. It removes the class of difference that comes from one group's simulation settings, integration timestep, or luckier hardware, and it is why the stability numbers mean something.

The survival test

Stability was measured by brute force. Twelve independent molecular dynamics simulations per model per condition, one million steps each, which is one nanosecond of simulated time. A run was terminated if any covalent chemical bond in the system stretched beyond 2 ångströms, a state the systems were not physically expected to reach. The result for each cell of the table is a ratio: trajectories completed over trajectories that broke.

The outcome, in the authors' summary:

The MACE architecture demonstrated the highest stability in our tests, successfully completing most MD trajectories. It only failed in the MAPbI3 system at 500 K, where it could not provide a single 1 ns long MD trajectory. The second most stable architecture, SO3krates, performed well in this scenario, successfully completing 10 out of 12 MD runs.

Sit with the shape of that. In the one system where the best-performing architecture failed completely, at zero out of twelve, the model that was second overall came through ten times out of twelve. The failure was not a general ranking of quality. It was system-specific, and it inverted.

Below the neural networks, the picture gets harsher. Among the kernel-based models sGDML held up best, performing well at 300 K across all challenges except the one with a deliberately incomplete training set. SOAP/GAP and FCHL19* were, in the paper's phrasing, "quite unstable," struggling across all challenges "with some simulations failing within just a few thousand steps." The authors are fair about why: those kernel models are normally trained inside active-learning workflows, and were not here, and they can still deliver dozens to hundreds of picoseconds in well-sampled regions near equilibrium.

The disconnect, stated plainly

Here is the sentence the whole report exists for. Discussing the neural networks trained on the unfolded alanine tetrapeptide and the perovskite:

Despite the fact that the mean and maximum force errors of these models were much lower than those of the kernel models, this did not necessarily translate to better stability of MD simulations at 500 K, for the case of SO3krates. This suggests a weak correlation between the accuracy of MLFFs on the test set and their reliability in actual simulations.

And the general form, two sentences later: "high aggregate accuracy on reference data does not necessarily translate to stable and reliable performance in practical MD simulations."

The authors then do the thing that makes this credible rather than merely provocative. They note that the two models with the best aggregate statistics were also consistently the most stable, and land on a formulation I would put on a wall: "overall accuracy is necessary for stability but not sufficient on its own."

That is a much more useful claim than "benchmarks are broken." The metric is not meaningless. It is a filter, not a ranking. A model with poor test-set accuracy will not give you stable dynamics; a model with excellent test-set accuracy might not either, and the metric cannot tell you which case you are in.

Why the metric misses it

The companion paper contains the cleanest mechanical illustration of how this happens, and it comes from the molecule-on-graphene system.

Both neural networks use a cutoff radius: beyond some distance, atoms simply do not see each other. At a molecule-to-surface separation of 8 ångströms, which is outside both cutoffs, the models predict zero interaction. Against the reference calculation, MACE underestimates the interaction energy at that distance by 2.9 kcal per mole and SO3krates by 4.4. At the adsorption minimum the underestimates are 3.5 and 4.9 kcal per mole.

Now the observation that makes it a lesson rather than an anecdote:

Interestingly, this underestimation of the molecule-surface interaction minimum aligns with the high accuracy in predicting the force acting on the molecule as a whole in the test set.

The models were accurate on the metric and wrong about the physics, in the same region, for the same reason. The reference data covered molecule-to-surface distances of 3 to 5 ångströms. Inside that window the curvature of the energy surface agrees with theory and the forces are right. The long-range attractive tail simply is not in the training set, is outside the cutoff, and contributes essentially nothing to the force error being reported. Averaging over sampled configurations cannot penalise a model for a region nobody sampled.

This is the failure mode I care most about, and the authors name it directly: "A significant risk can occur when MLFFs do not fail outright but produce seemingly reasonable yet ultimately incorrect results."

A simulation that explodes is a good day. You see it, you throw it away. A simulation that runs to completion, produces a smooth trajectory, and quietly understates an adsorption energy by 4 kcal per mole is the one that ends up in a paper. I spent my research life on microwave spectroscopy, where the entire discipline is built on the gap between a quantity you can measure precisely and the quantity you actually want; the instinct that transfers is that precision and correctness are separate properties, and only one of them announces itself.

Two honest complications

First, the models cannot tell you when they are out of their depth. The authors point out that kernel-based models admit relatively straightforward Bayesian uncertainty estimates, while "NN models such as MACE and SO3krates currently lack intrinsic mechanisms for estimating prediction uncertainty." The architectures that were most stable are the ones least able to flag their own extrapolation. That is a real trade, not a footnote.

Second, stability is not free. The timing table is blunt: SO3krates was fastest in production, "outperforming its NN competitor, MACE, by a factor of 6 to 25, depending on the system size." MACE needed 34.4 to 45.4 hours to generate a single nanosecond across the four challenges. So the most reliable model in the study is also, by a wide margin, the most expensive to run, and the practitioner choosing between them is trading compute budget against the probability that a trajectory survives.

What the authors actually conclude

Their overall verdict is less dramatic than the stability table and more useful. From the abstract of the molecular dynamics paper: "at the current stage of MLFF development, the choice of ML model is in the hands of the practitioner. When a problem falls within the scope of a given MLFF architecture, the resulting simulations exhibit weak dependency on the specific architecture used. Instead, emphasis should be placed on developing complete, reliable, and representative training datasets."

In other words: the architecture race that dominates the literature matters less, within scope, than the data. And the boundary of "within scope" is set by what you sampled, which is precisely what a test-set error cannot measure, because the test set is drawn from the same distribution as the training set.

The exception they flag is the one the graphene result illustrates: "long-range noncovalent interactions remain challenging for all MLFF models, necessitating special caution in simulations of physical systems where such interactions are prominent, such as molecule-surface interfaces."

Caveats, and they are large

Both papers state, twice and without hedging, that the findings "reflect the state of MLFF models as of October 2023." This field moves fast. Long-range corrections, foundation-model potentials trained on far broader chemistry, and uncertainty quantification for neural networks have all been active work since. Nobody should read the 2023 stability table as a current leaderboard, and the authors do not present it as one.

The models were also trained by their own developers on datasets chosen to stress them, in some cases deliberately incomplete. Several of the failures are the challenge working as designed rather than a defect in the architecture. And the two papers come from a collaboration that includes those developers, which cuts both ways: it means real expertise behind each model, and it means the participants had a stake in the outcome. The organisers' decision to run every simulation themselves is the structural answer to that, and it is why I trust the numbers.

What survives all of those caveats is not a ranking. It is an evaluation design, and it generalises past force fields to any learned model standing in for a physical calculation.

Three questions for a model that quotes an error

What does the model do outside the sampled region, and does the metric touch it? If the training data spans 3 to 5 ångströms and your simulation visits 8, the reported error is silent about the part you care about. Ask what the sampling covered before you ask how accurate it was. This is the same problem as a model that beats a baseline on the distribution it was fitted to.

Did anyone run the downstream task, or only score the model? The TEA Challenge is valuable because it did the expensive thing: twelve one-nanosecond trajectories per model per condition, on the organisers' hardware. Any evaluation that stops at the held-out set is measuring a proxy, and this study is direct evidence that the proxy and the task can disagree.

Can the model say it does not know? A silent failure is only silent because nothing raised its hand. Uncertainty estimation is not a nicety here; it is the difference between a wrong trajectory you catch and a wrong trajectory you publish. That is the same argument as auditing a fraction of a model's output against ground truth, and the same reason reproducibility has to be tested rather than assumed.

None of this is an argument against machine-learned force fields. They work, they are transforming what is computable, and the people who built them ran this evaluation on themselves and published the parts that did not flatter them. That is science behaving well. The lesson is narrower and it travels: a benchmark measures what it measures, and when a model is being used as a component of something larger, the only honest test is to run the larger thing and count what comes back.