ContextShift: A Controlled Benchmark for Context Dependence in Object Detection

2026-06-08

AI Summary

This work addresses the limited robustness of existing object detectors under contextual shifts and the inadequacy of conventional evaluation protocols in uncovering the underlying performance degradation mechanisms. The authors propose ContextShift, a controllable benchmark that treats context as an independent variable by systematically manipulating object–context relationships through geometric transformations and background replacement while preserving object appearance. They further introduce a continuous compatibility axis based on normalized pointwise mutual information (NPMI). Experiments on COCO 2017 reveal that contextual changes can increase miss rates by 227% and reduce detection counts by 44%, with optimal performance observed at moderate NPMI values. A novel context-aware data augmentation strategy derived from these insights substantially enhances model robustness.

Abstract

Modern object detectors achieve strong performance on standard benchmarks, yet their robustness to contextual variation remains insufficiently understood. Prior evaluations largely rely on aggregate metrics such as AP on uncontrolled distribution shifts, which can obscure how performance degrades under context change. We introduce ContextShift, a controlled benchmark that systematically manipulates object--context relationships while preserving object appearance. Built on COCO 2017, it isolates context as an independent variable through geometric transformations and synthetic and natural background substitutions, including a continuous compatibility axis based on normalized pointwise mutual information (NPMI). Across diverse detector architectures, we observe a consistent degradation pattern: false negatives increase by up to 227% and prediction volume decreases by up to 44%, while false positives remain stable or decline. This suppression behavior is not captured by aggregate metrics such as AP, which can mask substantial recall loss and changes in prediction dynamics. Further analysis suggests that degradation is driven less by reduced confidence than by a reduced formation of valid detection candidates. Moreover, performance along the statistical compatibility axis is non-monotonic, peaking at intermediate NPMI and degrading toward both extremes, indicating that statistical co-occurrence does not correlate linearly with effective visual context. Finally, we show that context-aware augmentation improves robustness: every augmented variant outperforms the dataset-only baseline on both original and manipulated test images, partially recovering performance lost to prediction-suppression failures by exposing models to object--context decoupling during training.

Problem

• context dependence
• object detection
• distribution shift
• robustness
• benchmark

Innovation

• ContextShift
• context dependence
• object detection robustness
• normalized pointwise mutual information
• context-aware augmentation