Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval

2026-07-21

AI Summary

This work addresses the unreliability and instability in zero-shot video moment retrieval caused by modality gaps and mismatches in linguistic style. To this end, it proposes a video self-similarity–based moment proposal and scoring mechanism that generates robust temporal candidates without relying on external queries. Furthermore, it introduces, for the first time, a query-aware multimodal large language model (MLLM) during inference to enable fine-grained cross-modal alignment. This approach effectively circumvents noisy matches between queries and individual frames or captions, significantly improving both retrieval accuracy and stability across multiple zero-shot benchmarks and achieving state-of-the-art performance.

Abstract

Zero-shot video moment retrieval aims to overcome the limitations of traditional approaches that require large-scale datasets annotated with text and its relevant temporal spans. Despite advances in pre-trained vision-language models and multimodal large language models, existing ZMR methods still heavily depend on query-to-video content similarity, making them vulnerable to modality and language-style gaps. These gaps lead to unreliable span proposals and unstable moment retrieval results. To address this issue, we propose Self-Similarity-based Moment Proposal and Scoring that instead exploits intrinsic relationships within videos, enabling robust span generation and scoring. By deriving self-similarity only from the video content, we circumvent the noisy and mismatched patterns of query-frame or query-caption similarities, thereby mitigating both modality and language-style gaps. Furthermore, we introduce a query-aware MLLM-based reasoning stage to further sharpen alignment between text and video. Extensive experiments demonstrate that Self-SiMS achieves state-of-the-art performance across ZMR benchmarks.

Problem

• Zero-shot video moment retrieval
• modality gap
• language-style gap
• moment retrieval
• video-text alignment

Innovation

• zero-shot video moment retrieval
• self-similarity
• modality gap
• language-style gap
• multimodal large language model