Benchmark: method, results and limits
What we measured on 2026-10-20, how, and what it does not show. We keep the earlier verdict next to the new one.
Method
The test set (test2) has 1,000 French addresses, a mix of clean and deliberately degraded ones (abbreviations, typos, missing postcode, free text, reversed order, PO boxes, hamlets, overseas). It was drawn with a fixed seed, shares no address with the earlier sets, and was frozen by its SHA-256 hash (9de1b2df…) before any run.
The criterion was chosen, and the pass/fail rules recorded and committed, before the run. The set was used once, with no tuning afterwards. It is now consumed: any new verdict needs a new set and a newly recorded protocol.
An error is an address with no answer, or whose point lies 50 m or more from the reference point. The reduction in errors is one minus the errors of Addrova divided by the errors of the BAN alone, counted on the same 610 noisy addresses. Intervals are 95 % bootstrap intervals from 10,000 paired resamples.
A confident false positive is an accepted answer (raw score at or above 0.76) that is wrong by more than the category tolerance or lands in the wrong commune.
Results against the rules fixed beforehand
| Criterion | Result | Met |
|---|---|---|
| Reduction in errors on noisy addresses of at least 50 % (point estimate) | 56.4 % [33.3 ; 72.9]: 24 errors against 55 for the BAN alone | Yes |
| Confident false positives at most 1 % | 0.22 % (95 % upper bound 0.80 %) | Yes |
| p95 latency at most 150 ms | 23 ms | Yes |
| Calibration error (ECE) at most 0.05 | 0.005 | Yes |
| Clean addresses: hit rate ≥ 97 % and answer rate ≥ 99 % (supplementary) | 100 % and 100 % | Yes |
The criterion “at most 3 points below the leading commercial provider” was not evaluated: no comparison with a commercial provider has been run.
Verdict: GO, outside any comparison with Google
By the rules fixed beforehand, the result is a GO. The point estimate of the error reduction is above 50 %, but the interval is wide, from 33 % to 73 %, because the BAN alone makes few errors on this set. In 75 % of resamples the reduction is 50 % or more.
We do not claim Addrova is better than any other service. We report these figures, on this test set, against the BAN alone.
Limits
- The reference point is the BAN’s own point, which biases the comparison in favour of the BAN and of engines built on it.
- One country (France). Other countries are not measured here and carry no figure on this site.
- No comparison with a commercial provider: a GO here is a GO outside that comparison.
- Latency was measured locally, one request at a time, on a shared machine: it is not a service-level commitment.
- The interval of the reduction is wide when the reference makes few errors; read the interval, not only the point estimate.
- Full reports and the protocol are kept in the project’s benchmark folder; their public release is not decided yet. [À VÉRIFIER]
History: the first verdict, 2026-10-19
The first measurement used a different test set and a stricter criterion: a gain of at least 10 points on noisy addresses. The result was a PIVOT (grey zone): the gain was +7.7 points [+5.2 ; +10.2], and confident false positives were 13 out of 871 accepted answers, 1.49 %, above the 1 % target.
That test set was consumed and its verdict stands as recorded. The second measurement is not a re-run of it: it uses a new set, drawn with a new seed, and a different criterion chosen afterwards (errors removed).
What comes next
Extend the measurement to other countries as they are ingested, each with its own frozen set and recorded protocol, and run a comparison with commercial providers only with their terms allowing it.