Corvus ISR Publishes Synthetic Scene Benchmark for Tracker Accuracy

Corvus ISR tracker benchmark matrix (seed 1337) — The published matrix — every row reproducible. Source: corvusisr.com/benchmark

Corvus ISR, known for its wide-area motion imagery (WAMI) exploitation products, has released a detailed public tracker benchmark comparing two distinct tracking models. This benchmark uses a fixed-seed synthetic scene with perfect ground truth, ensuring that the results are purely about the tracker’s performance rather than external variables. The scene runs for 20 seconds of warm-up followed by 120 seconds of measured data, with all aspects such as sensor modeling and detection generation kept byte-identical, except for the tracking algorithms themselves.

The first model, v1, employs a greedy nearest-neighbour approach, featuring a two-pass greedy association, constant velocity prediction, and a fixed 2-second coasting period. It serves as the baseline, representing a deliberately simple but functional tracking method. The second model, v2, introduces a confirmed-track auction mechanism, incorporating three-tier auction association, velocity consistency gating, noise-scaled reservation, and confidence-decayed coasting — all aimed at improving the tracking accuracy and robustness under challenging conditions.

Results from the benchmark reveal significant improvements with v2: in a scenario with 150 movers at 2 frames per second, ID switches per minute dropped from 2,042 to 1,183, a 42.1% reduction. Increasing density to 400 movers saw ID switches decrease from 14,032 to 8,040, a 42.7% decrease. Under degraded conditions such as lower frame rates, occlusion, and noise, the number of identity errors also declined substantially, demonstrating the effectiveness of the newer tracking approach.

It’s important to note that the published numbers strictly quantify errors, with the ID switch metric counting every change in the assigned identity to a ground-truth object — including fragmentations and re-acquisitions, making it stricter than typical MOT challenge standards. Publishing such failure numbers emphasizes transparency; even with these improvements, thousands of identity errors still occur, highlighting the ongoing challenges in tracker development.

Why emphasize failures? Because synthetic scenes provide perfect ground truth, allowing for precise measurement rather than marketing claims. These results serve as a benchmark for future tracker development, demanding that every new model be individually tested and published against the same fixed seed. This approach ensures that progress is measurable and transparent, reinforcing the importance of honest performance reporting in AI development.

From an engineering perspective, v2 runs efficiently, averaging around 1.2 milliseconds per sensor tick with a dense scene of 400 objects, with the worst case at about 5 milliseconds — well within real-time constraints for browser-based visualization. Anyone interested can reproduce the scene live and run the benchmark themselves, without signup or NDA, to see the results firsthand. The development of v2 was guided by an AI executor under a written contract and independently reviewed, ensuring robustness and transparency in its methodology.

All of these results are generated in a fully synthetic environment — no real persons, vehicles, or locations are involved; every pixel is artificially created. This methodology underscores the importance of synthetic data for rigorous testing of AI trackers, providing a controlled environment where ground truth is perfect and performance metrics are definitive. Readers interested in how these benchmarks are constructed and measured are encouraged to explore the public benchmark and try it out themselves.