🤖 AI Summary
This work addresses the neglect of structured relationships among objects within images in open-vocabulary object detection by explicitly modeling semantic and spatial interactions between candidate regions and contextual objects through scene graphs—a first in this domain. The proposed framework integrates a relation-aware attention module with a scene-text alignment branch, jointly leveraging visual relational cues and linguistic semantic knowledge. It further incorporates knowledge distillation and alignment strategies with vision–language models to enhance generalization. Evaluated on the COCO and LVIS benchmarks, the method achieves significant improvements in average precision (AP) for novel categories, outperforming existing open-vocabulary object detection approaches.
📝 Abstract
Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.