🤖 AI Summary
This study addresses the widespread misuse of confidence scores from open-vocabulary detectors—such as Grounding DINO, OWLv2, and SAM3—as proxies for object visibility, when in fact these scores reflect category presence rather than the actual visibility of a specific instance. By constructing a ground-truth visibility benchmark using a geometric segmentation oracle and evaluating across multiple simulated environments and real-world video data, the work systematically audits current models and reveals, for the first time, that detector confidence remains high even when only 1/8 of an object is visible. This fundamental mismatch leads to a nearly tenfold underestimation of active perception performance. The authors argue that confidence-based evaluation and gating mechanisms are inherently biased and advocate instead for object-anchored visibility signals. To support future research, they release the first controllable occlusion benchmark dataset.
📝 Abstract
Occlude a named object until about an eighth of it remains visible, and an open-vocabulary detector's confidence that the object is present barely changes; as the clutter around it grows the confidence can even rise. On real video the detector still reports the object present in 99% of occluded frames, on another instance of the same category. This matters because that confidence is widely read as a visibility signal, used to threshold detections, evaluate open-vocabulary detectors, ground language, retrieve instances, and gate active perception. We audit whether it reflects occlusion by pairing every view with a geometry-segmentation oracle that gives detector-free ground-truth visibility. As true visibility falls from every scene to one in eight, the confidence stays nearly constant and uncorrelated with visibility, and the detector reports the target present in about nine of ten scenes, firing on same-category distractors: it signals that the category is present somewhere, not that the specific target is visible. The failure holds across three detectors (Grounding DINO, OWLv2, and Segment Anything Model 3), nine object categories, two simulators with different renderers and object sets, built and natural occlusion, and real video. Two consequences follow: a confidence-based metric understates the value of resolving occlusion by about ten times (8 against 88 points in our active-perception setting), and a confidence-based gate fires exactly when the object is hidden. No single-view signal we tried, including a realizable localization check, flags the occlusion, because the occluders sit where the target is. We connect the effect to detector miscalibration and object hallucination, release the controlled benchmark, and recommend target-grounded signals for gating and evaluation.