NADIR.
Validation

How we will measure error

A forensic method is only as useful as its known error. We are fixing how we will measure ours before we look at any results.

Benchmark in progress. No forensic accuracy figures exist yet, and none appear on this site. The protocol below is pre-registered in NADIR-TR-2026-03 (draft for founder review). Acceptance thresholds are placeholders until the team fixes them.

Rule

Synthetic data is for debugging only. The demo never gets ahead of the benchmark: no capability is shown publicly before the benchmark stage that measures it has been run and written up.

Benchmark plan

Three stages, each harder and more realistic than the last. A stage starts only after the previous stage's report is written, and results from different stages are never pooled.

StageGround truthWhat it testsStatus
1KITTI, nuScenes, Waymo Open: real driving video with lidar and GNSS/INS posesGeometry and tracking on real imagery with factory-calibrated camerasPlanned
2Instrumented closed-track runs (target venue: Mcity, Ann Arbor) with RTK-GNSS/INS on every vehicleThe primary error-rate measurement: consumer cameras, varied mounts, real timing jitter, near-miss and AEB-style scenariosPlanned
3Blind comparison on closed cases with credentialed (ACTAR-accredited) reconstruction expertsAgreement with the current expert standard of care on real crashesPlanned

Stage 1 uses all three public datasets. Mcity is a target venue, not an agreement.

Metrics we will publish

Every metric is reported for every stage, including the ones that look bad. Means come with clip-level bootstrap confidence intervals.

Speed error
Estimated minus true speed (m/s) for the recording vehicle and each actor at fixed query times: impact, braking onset, and 1, 2 and 3 s before. MAE, RMSE, bias, median and 95th percentile.
Position error
Road-plane distance between estimated and true position, with no after-the-fact alignment to ground truth; plus the error in the gap between vehicles.
TTC error
Error in time to collision, TTC = −d / ḋ, at each query time while the vehicles are closing.
Event timing
Error in milliseconds for braking onset, first visibility and impact.
Interval coverage
How often the true value falls inside the stated 50 / 80 / 90 / 95% intervals, with mean interval width so coverage cannot be bought with vague intervals.
Envelope calibration
For the sensor-suite counterfactual: Brier score, expected calibration error and a reliability diagram against observed warnings and interventions on instrumented test vehicles.
Decline rate
How often the engine refuses to answer (for example, too few scale cues), broken down by reason.

Definitions

Position error at query time ti, in the road plane, with no after-the-fact alignment to ground truth: the pipeline must recover scale and pose itself.

eipos=‖𝐩^(ti)−𝐩(ti)‖2e^{\mathrm{pos}}_i = \bigl\lVert \hat{\mathbf{p}}(t_i) - \mathbf{p}(t_i) \bigr\rVert_2

Time to collision while closing, compared between estimate and ground truth at each query time.

TTC(t)=−d(t)d˙(t),d˙(t)<0\mathrm{TTC}(t) = -\frac{d(t)}{\dot{d}(t)}, \qquad \dot{d}(t) < 0

Coverage of a stated interval Ĉi at level 1 − α. Under-coverage counts as worse than over-coverage, because it overstates certainty in a legal setting.

cov1−α=1n∑i𝟏[yi∈C^i]\mathrm{cov}_{1-\alpha} = \frac{1}{n} \sum_{i} \mathbf{1}\bigl[\, y_i \in \hat{C}_i \,\bigr]

Calibration of the detection envelope against observed warnings and interventions oi ∈ {0, 1} on instrumented test vehicles, over B equal-mass bins.

Brier=1n∑i(pi−oi)2,ECE=∑b=1B|Ib|n|o‾b−p‾b|\mathrm{Brier} = \frac{1}{n} \sum_{i} (p_i - o_i)^{2}, \qquad \mathrm{ECE} = \sum_{b=1}^{B} \frac{|I_b|}{n} \Bigl| \bar{o}_b - \bar{p}_b \Bigr|

Formulas, splits, blinding and threats to validity are in the protocol report.

Current status

ItemStatus
Protocol written (NADIR-TR-2026-03)Draft for review
Acceptance thresholds fixedNot yet
Stage 1 runsNot started
Stage 2 track agreement and runsNot started
Stage 3 expert partners and casesNot started
Forensic accuracy figuresNone published

Software tests

Unit tests check that code does what it was written to do. They say nothing about forensic accuracy. Counts below were taken by running the suites on 2026-10-31.

SuiteResultScope
NADIR_PERCEPTION_OS/tests41 passedDetection/depth pipeline plumbing, planar ground, lane-residual fusion, filter state, recorder
NADIR_SIMULATION/tests48 passed, 9 failed, 4 not collectedPrior drift product simulator; failures and collection errors in a clean environment without its full dependencies (e.g. httpx). Not forensic.

There is no test suite for the forensic pipeline yet because the forensic layers (EDR fusion, counterfactual) are not built.