Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrieval

2025-08-15

AI Summary

Existing text-driven video moment retrieval (VMR) methods encode entire videos uniformly, introducing substantial irrelevant content that degrades cross-modal alignment and hinders optimization. To address this, we propose a novel “denoise-then-retrieve” paradigm. Specifically, we introduce the first text-conditioned video denoising mechanism, which jointly leverages cross-attention and structured state-space modules to identify noise segments; text reconstruction feedback then enables dynamic noise filtering and query embedding distillation. Subsequent VMR is performed on the purified multimodal representations. Our approach is plug-and-play—compatible with mainstream VMR architectures without architectural modification. Extensive experiments demonstrate state-of-the-art performance on Charades-STA and QVHighlights, achieving significant gains in both retrieval accuracy and robustness over prior methods.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Current text-driven Video Moment Retrieval (VMR) methods encode all video clips, including irrelevant ones, disrupting multimodal alignment and hindering optimization. To this end, we propose a denoise-then-retrieve paradigm that explicitly filters text-irrelevant clips from videos and then retrieves the target moment using purified multimodal representations. Following this paradigm, we introduce the Denoise-then-Retrieve Network (DRNet), comprising Text-Conditioned Denoising (TCD) and Text-Reconstruction Feedback (TRF) modules. TCD integrates cross-attention and structured state space blocks to dynamically identify noisy clips and produce a noise mask to purify multimodal video representations. TRF further distills a single query embedding from purified video representations and aligns it with the text embedding, serving as auxiliary supervision for denoising during training. Finally, we perform conditional retrieval using text embeddings on purified video representations for accurate VMR. Experiments on Charades-STA and QVHighlights demonstrate that our approach surpasses state-of-the-art methods on all metrics. Furthermore, our denoise-then-retrieve paradigm is adaptable and can be seamlessly integrated into advanced VMR models to boost performance.

Problem

• Filters irrelevant video clips for better multimodal alignment
• Purifies video representations to improve moment retrieval accuracy
• Enhances existing VMR models with adaptable denoising-retrieval paradigm

Innovation

• Text-Conditioned Denoising filters irrelevant video clips
• Cross-attention and state space blocks create noise mask
• Text-Reconstruction Feedback aligns purified video-text embeddings