FATE: Frame-Level Audio-Visual Temporal Embedding

2026-08-02

AI Summary

Existing audio-visual models struggle to simultaneously achieve semantic understanding and temporal alignment: embedding-based approaches often lose fine-grained temporal details, while synchronization-focused methods lack rich semantic representation. To address this limitation, this work proposes FATE, a novel framework that preserves and strictly aligns frame-level audio-visual sequences on the physical timeline. FATE jointly models “what sound occurs” and “when it happens” within a unified, reusable multimodal embedding space by leveraging cross-video semantic contrastive learning and intra-video temporal contrastive learning. This approach is the first to integrate semantic content and precise temporal synchronization within a single embedding space, overcoming a key bottleneck in prior models. Experiments demonstrate that FATE significantly outperforms state-of-the-art baselines across three tasks, matches fully supervised methods in zero-shot event localization, and achieves the highest correlation with human judgment in generative evaluations.

Abstract

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at \texttt{https://github.com/guankaisi/FATE}.

Problem

• audio-visual alignment
• temporal embedding
• semantic understanding
• frame-level synchronization

Innovation

• audio-visual alignment
• temporal embedding
• frame-level synchronization
• contrastive learning
• zero-shot event localization