REZE: Recognition-Based Zero-Shot Extraction for Video Temporal Grounding

2026-08-05

AI Summary

This work addresses the instability of existing zero-shot video temporal grounding methods that directly rely on large language models to generate timestamps. To overcome this limitation, the authors propose a training-free zero-shot framework that first segments the input video into short clips, then leverages a vision-language model to produce clip-level confidence scores, and finally aggregates these scores into precise temporal intervals using a deterministic post-processing algorithm. By externalizing temporal aggregation from the model itself, the approach uniformly supports diverse tasks—including single- and multi-moment retrieval as well as highlight detection—and is compatible with various backbone architectures. On QVHighlights, the method achieves 40.32 mAP for moment retrieval, 44.18 mAP and 73.41 HIT@1 for highlight detection, substantially outperforming all existing zero-shot approaches and even surpassing several fully supervised state-of-the-art methods.

Abstract

Video temporal grounding (VTG) refers to the task of identifying the time interval in a video that corresponds to a given natural-language query. A common zero-shot strategy asks a large vision-language model (VLM) to generate the start and end timestamps directly, so the result depends heavily on the design and training of the model, and grounding accuracy differs widely from one VLM to another. We therefore propose REcognition-based Zero-shot Extraction (REZE), a simple training-free method that splits the video into short clips, asks the model for a clip-level confidence score for the query, and uses a deterministic algorithm to convert the resulting score curve into the output required by the task. Because temporal aggregation is performed outside the model, REZE adapts to different task outputs, from single- and multi-interval moment retrieval to highlight detection. On QVHighlights, REZE improves the best reported training-free moment-retrieval mAP from 38.23 to 40.32, while on highlight detection it reaches 44.18 mAP and 73.41 HIT@1, establishing a new state of the art among training-free methods. Its HIT@1 also outperforms all fully supervised SoTAs on the QVHighlights test split. We evaluate REZE on seven backbones from three model families. On Charades-STA and QVHighlights, it outperforms direct timestamp generation in every available comparison. We further observe that with REZE an earlier-generation model can approach the native performance of a newer model in its family.

Problem

• video temporal grounding
• zero-shot learning
• vision-language model
• temporal localization
• moment retrieval

Innovation

• zero-shot
• video temporal grounding
• recognition-based extraction
• vision-language model
• temporal aggregation