Structured Spatio-Temporal Evidence Graphs for Open-Vocabulary Object Retrieval in Videos

2026-09-20

AI Summary

该研究针对视频中开放词汇对象检索问题，提出了一种结构化时空证据图STEG-OVR方法，通过将查询分解为多个槽位来激活相应的检索通道，从而提高检索准确性。

Abstract

Open-vocabulary object retrieval in videos requires answering free-form object queries under bounded query-time cost. Existing index-based systems typically store independent frame-level regions and retrieve them with vision-language similarity, which is effective for appearance queries but mismatched with predicates whose evidence is temporal or relational, such as stopped state, scene-region occupancy, persistence, and object interactions. We identify this gap as an evidence-unit mismatch: the query is expressed over tracklets or object tuples, while the index stores isolated boxes. To address it, we propose STEG-OVR, a structured spatio-temporal evidence graph for open-vocabulary object retrieval. STEG-OVR represents persistent objects as tracklet nodes and temporally compatible object pairs as relation edges, storing appearance, motion, scene occupancy, relative geometry, velocity compatibility, and symbolic relation evidence. A query is decomposed into entity, state, scene, temporal, and relation slots, which activate only the corresponding retrieval channels before soft score fusion and fixed-budget consistency verification. Diagnostic experiments on three object-centric settings show AP improvements from 0.0882 to 0.2073 on Beach, from 0.0834 to 0.1505 on Shibuya, and from 0.701 to 0.743 in a LOVO-style comparison. The gains are strongest for stopped-state and scene-region-occupancy queries, while sustained relations remain sensitive to tracking continuity and predicate calibration.

Problem

• open-vocabulary object retrieval
• temporal evidence
• relational evidence

Innovation

• Structured Spatio-Temporal Evidence Graphs
• Open-Vocabulary Object Retrieval
• Tracklet Nodes
• Relation Edges
• Evidence-Unit Mismatch