Explaining Object Detectors via Collective Contribution of Pixels

2024-12-01

arXiv.org

AI Summary

Existing interpretability methods for object detection predominantly rely on single-pixel attribution, failing to capture the joint influence of multi-pixel collaborations on both bounding box localization and class prediction—thus overlooking compositional cues or introducing spurious correlations. To address this, we introduce Shapley interaction values to object detection interpretation for the first time, proposing the first end-to-end differentiable framework that explicitly models high-order cooperative effects among pixel groups. Our approach jointly models feature-space perturbations and detection output sensitivity to simultaneously quantify individual pixel contributions and higher-order interactions. Extensive experiments on COCO and other benchmarks demonstrate that our method significantly outperforms state-of-the-art interpretability baselines. Both qualitative visualizations and quantitative metrics confirm its superior ability to localize discriminative visual regions accurately. The source code will be made publicly available.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Visual explanations for object detectors are crucial for enhancing their reliability. Since object detectors identify and localize instances by assessing multiple features collectively, generating explanations that capture these collective contributions is critical. However, existing methods focus solely on individual pixel contributions, ignoring the collective contribution of multiple pixels. To address this, we proposed a method for object detectors that considers the collective contribution of multiple pixels. Our approach leverages game-theoretic concepts, specifically Shapley values and interactions, to provide explanations. These explanations cover both bounding box generation and class determination, considering both individual and collective pixel contributions. Extensive quantitative and qualitative experiments demonstrate that the proposed method more accurately identifies important regions in detection results compared to current state-of-the-art methods. The code will be publicly available soon.

Problem

• Explains object detectors via collective pixel contributions
• Captures both individual and collective pixel influences
• Provides explanations for bounding box and class decisions

Innovation

• Game-theoretic method using Shapley values
• Captures individual and collective pixel contributions
• Explains bounding box localization and class determination