🤖 AI Summary
Temporal and spatial localization of visual queries in long videos remains challenging due to small object sizes, occlusions, appearance variations, and cluttered backgrounds.
Method: We propose a training-free, strong baseline that leverages pre-trained ViT/CLIP for region representation extraction, integrating YOLO-style detection, inter-frame cosine similarity matching, and bidirectional optical flow tracking—augmented with object-aware fine-grained relocalization and visual query enhancement. This design significantly improves robustness to partial visibility and dynamic appearance changes.
Contribution/Results: On the Ego4D VQ 2D Localization benchmark, our method achieves a 49% relative improvement in spatiotemporal mean average precision over prior supervised approaches. It establishes the first high-performance, training-free baseline for video grounding, introducing a novel zero-shot paradigm for visual query localization in long videos.
📝 Abstract
We present Relocate, a simple training-free baseline designed to perform the challenging task of visual query localization in long videos. To eliminate the need for task-specific training and efficiently handle long videos, Relocate leverages a region-based representation derived from pretrained vision models. At a high level, it follows the classic object localization approach: (1) identify all objects in each video frame, (2) compare the objects with the given query and select the most similar ones, and (3) perform bidirectional tracking to get a spatio-temporal response. However, we propose some key enhancements to handle small objects, cluttered scenes, partial visibility, and varying appearances. Notably, we refine the selected objects for accurate localization and generate additional visual queries to capture visual variations. We evaluate Relocate on the challenging Ego4D Visual Query 2D Localization dataset, establishing a new baseline that outperforms prior task-specific methods by 49% (relative improvement) in spatio-temporal average precision.