How do you score a bomb detector?

In 2022 the European Commission put a sentence in a funding document that anyone who has worked Counter-IED will recognize:

IED and landmine detection has been a research topic for many years. However, progress is hindered by the lack of standardised benchmarks.

That line sits in the European Defence Fund's 2022 work programme. The call behind it has a €25 million budget, split into €20 million for teams building detection systems and €5 million for one organizer to run the test. The organizer is HiTDOC, a five-member consortium coordinated by Sweden's defence research agency FOI, with Fraunhofer, Spain's INTA, France's LNE and the Dutch TNO. Its EU factsheet lists about €4.99 million and 48 months. Fraunhofer ICT says the project runs from December 2023 to November 2027. Four teams compete: AIDEDex, CONVOY, DeterMine and TICHE.

The scores aren't public, as far as I can find. But the rulebook is. Annex 4 of the call document is a preliminary evaluation plan, and it contains this admission:

Since this is the first challenge in this domain, defining the metric and protocols is also an objective in itself.

So the question in the title is not rhetorical. It's a funded work item.

What the test looks like

Each system pairs a main ground vehicle with at least one accompanying ground robot or aerial drone, all carrying sensors. The team has to get from a start area to a target location through a zone seeded with simulated devices, which the plan describes as "real-scale fully functional devices" that stay harmless by "containing only small explosive charges or smoke bombs." The focus is static devices, in subsurface, surface and side-attack positions, both command-operated and victim-operated. Sensor data is recorded so software can be improved between field campaigns.

The plan then lists what gets scored:

• Avoidance: hits on the main vehicle per kilometer of track. Hits on the accompanying robots and drones are counted separately, as a secondary metric.
• Detection and mapping: the number of errors, meaning false alarms plus missed detections, per square kilometer.
• Type identification: classification error rate. The baseline splits IEDs into command-operated versus victim-operated, and landmines into anti-personnel versus anti-tank.
• Feature estimation: average percentage error, with charge size as the baseline feature.

Notice what isn't on that list: a bare probability of detection. The first metric is an outcome: did the vehicle get hit? The second charges you for both kinds of mistake at once.

Four design choices worth stealing

Run it with the detector off. The plan calls for runs "with and without the help of the automatic hidden threat detection." That's a control group. It measures what the system adds to a human pilot, not what the system can do in isolation. If you take one idea from this report, take that one. A detector that finds things a good driver would already have avoided hasn't bought much.

Don't let detours improve the score. Teams may leave the recommended path to avoid a threat, but the distance used for the metric stays the organizers' track: "Possible detours do not lead to use a longer track length for computing the metric." Wander around all day and the denominator doesn't grow.

Keep some of the answers hidden. Ground truth for at least one test zone is withheld until it's used in the online tests. Those tests send the same data to all teams at the same time, with outputs due a few days later, and the rules say participants "should not look at the data content until completion of its processing." That's how you stop a team from tuning to the answer key, a problem I covered from the data side in the report on data leakage.

Position counts. A detection that doesn't match ground truth "within a certain tolerance threshold" is a false alarm, and a real device nobody matched is a missed detection. Follow that logic and a correct call in the wrong place costs you twice. That's my reading of the plan's definitions, not a line it spells out, but it's the right incentive. A marker ten meters from the device doesn't protect the vehicle.

Where the score can still mislead

One error count treats two very different mistakes as equal. In the detection metric, a false alarm and a miss each add one to the total. On a route they aren't equal. A false alarm stops the convoy and costs time. A miss can cost the vehicle. I wrote in the mine detector report that those two rates are coordinates on one curve, so a single sum can hide which way a system was tuned. The avoidance metric partly covers this, because a miss can turn into a hit. And the detailed evaluation plans may report the two separately. I haven't seen them.

The range isn't a stranger. Organizers provide a map of the recommended path in advance, run reconnaissance of the track before the tests, and allow several runs through the same range so systems can use change detection. That's a fair model of a road you drive every day. It's a generous model of a road you've never seen. Both situations are real, and the score belongs to the first one.

The test can't fight back. The devices are static and were placed by the organizers. The plan allows that moving, UAV-borne devices "can be considered for the last years of the challenge." Nothing in a benchmark adapts to your sensor the way a real adversary does. I made the same point about jammers in the report on the two clocks: the threat keeps moving after the hardware is judged. A benchmark is a dated photograph of the problem, and that's still far better than no photograph.

The questions change between years. The plan calls the first campaign "a dry-run phase, where some adaptation of the evaluation protocols might be needed before delivering meaningful measurements." That first field trial ran 23 to 27 September 2024 in Sweden. For the second, in September 2025 at FOI's site in Grindsjön, CONVOY's coordinator GMV says evaluations "prioritized early detection and subsequent classification, as opposed to the threat-avoidance trials carried out in 2024." That's sensible for a young program, but it means a 2024 result and a 2025 result may not be the same measurement. The plan anticipates this by setting some data aside "to enable more meaningful performance comparisons across years."

Field tests don't repeat. The call says it plainly: "field tests cannot be repeated at will and are not perfectly reproducible, especially for detection systems that involve artificial intelligence." Hence the two-track design. Online tests on recorded data are repeatable but a step removed from the dirt. Field tests are real but happen once. You need both, and a claim that cites only one of them is telling you half.

You may never see a league table. Under the plan, teams "can communicate on their own results and methods." Challenge-level results go through the organizers, then to the teams for comment, and to the granting authority "for approval" before publication. That's normal for defense work. It also means any number you do see may be the one a team chose to share. Fraunhofer ICT called the 2024 trial "very successful" while noting the systems were still in development. Read that as a statement about the trial, not about detection performance.

This has been done before, in a calmer setting

The plan's "first challenge in this domain" is fair for IEDs on this scale, but buried-object testing has a useful ancestor. The U.S. Standardized UXO Technology Demonstration Sites Program, at Aberdeen and Yuma Proving Grounds, was built to compare sensors that hunt unexploded ordnance. Its 2006 fact sheet describes a calibration area, a blind grid where each square holds nothing, a munition or a clutter item, and an open field with challenges such as electrical lines, wet areas, steel fencing and wooded terrain.

Two features stand out. Scoring was automated against ground truth, because, in the program's words, it "eliminates potentially subjective interpretation of performance." And detection and false alarms were reported as receiver-operating characteristic curves, with false alarms on known clutter split from background alarms on nothing at all. The reports were posted to the program's website.

That problem was easier in one key way: unexploded ordnance doesn't redesign itself. But it shows what good reporting looks like. Show the curve, separate the types of false alarm, publish it. The same fact sheet also warned that results vary with terrain, geology, vegetation and weather, which is the point of the report on what soil does to detection.

Questions to ask any detector claim

• Who placed the targets, and did the team know where? A blind test and a demonstration are different things. I went through this in what a bomb robot demo doesn't prove.
• What's the false alarm rate per unit of area or route? A detection rate quoted alone is the easy half.
• What did it add over a trained operator? Ask for the with-and-without comparison.
• Was the route seen beforehand? Prior reconnaissance and repeat passes change what the score means.
• How many targets, how many runs, and what significance test? The EU call asks for significance tests on the measurements. A vendor should be able to answer the same question.
• What environment? One range, one season, one soil is one data point.

What I could not confirm

I've read the preliminary plan, not the detailed ones. Annex 4 dates from 2022, and it says more detailed plans will be produced for each campaign. Metrics may have changed. Everything above describes the published preliminary plan, not necessarily what was scored in Sweden.

I found no published scores. I found no detection rates, false alarm figures or rankings from the 2024 or 2025 campaigns. That's a statement about my search, not proof that none exist.

I couldn't confirm a 2026 campaign. GMV said in October 2025 that two more campaigns were planned over the next two years. I found no public record of the third.

Some sources are thin. I read only the abstract of the 2025 Fraunhofer conference paper on HiTDOC. The 2025 campaign details come from GMV, a participant, not the organizers. For the UXO program I read the fact sheet, not individual scoring records, since DTIC was down for maintenance when I checked, and that program's status today is outside this report.

The signal

There's no honest public answer yet to "which IED detector is best," and the EU's own documents say why: the field lacked standard benchmarks, and field tests can't be repeated at will. The plan it published is still worth reading, because it scores outcomes, runs a control, hides the answers and charges for false alarms. Its limits are the limits of any benchmark: the range is known, the devices don't adapt, and one total can hide the trade between a stop and a hit. When someone hands you a detection number, ask what test produced it. If they can't answer, you don't have a number yet.