The Devil is in the Spurious Correlation: Boosting Moment Retrieval via Temporal Dynamic Learning

2025-01-13

AI Summary

In temporal video grounding, textual queries often spuriously correlate with background frames, leading to inaccurate moment localization. To address this, we propose a Temporal Dynamic Learning (TDL) framework that constructs diverse contextual representations via dynamic video synthesis and—crucially—aligns text queries to temporally dynamic visual representations for the first time, enabling decoupled semantic and temporal modeling and strengthening causal associations between queries and ground-truth moments. Our method integrates a Transformer-based multimodal architecture, video segment mixing and synthesis, temporal convolution with dynamic attention, and joint cross-modal contrastive learning with fine-grained alignment loss. We achieve state-of-the-art performance on QVHighlights and Charades-STA, with significant improvements in IoU@0.5 and mean Average Precision (mAP). Ablation studies confirm the effectiveness and complementary nature of dynamic synthesis and dynamic alignment strategies.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Given a textual query along with a corresponding video, the objective of moment retrieval aims to localize the moments relevant to the query within the video. While commendable results have been demonstrated by existing transformer-based approaches, predicting the accurate temporal span of the target moment is currently still a major challenge. In this paper, we reveal that a crucial reason stems from the spurious correlation between the text queries and the moment context. Namely, the model may associate the textual query with the background frames rather than the target moment. To address this issue, we propose a temporal dynamic learning approach for moment retrieval, where two strategies are designed to mitigate the spurious correlation. First, we introduce a novel video synthesis approach to construct a dynamic context for the relevant moment. With separate yet similar videos mixed up, the synthesis approach empowers our model to attend to the target moment of the corresponding query under various dynamic contexts. Second, we enhance the representation by learning temporal dynamics. Besides the visual representation, text queries are aligned with temporal dynamic representations, which enables our model to establish a non-spurious correlation between the query-related moment and context. With the aforementioned proposed method, the spurious correlation issue in moment retrieval can be largely alleviated. Our method establishes a new state-of-the-art performance on two popular benchmarks of moment retrieval, ie, QVHighlights and Charades-STA. In addition, the detailed ablation analyses demonstrate the effectiveness of the proposed strategies. Our code will be publicly available.

Problem

• Video Moment Retrieval
• Temporal Localization
• Text-Video Alignment

Innovation

• Temporal Variation Learning
• Video Mixing
• Moment Retrieval