LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

2026-10-05

AI Summary

This study addresses the reliance of existing audio-visual self-supervised learning methods on complex mechanisms and the absence of a concise, collapse-resistant training framework. We propose the first minimalist architecture based on the LeJEPA objective, employing an early-fusion ViT encoder with modality dropout as the core mechanism for implicit cross-modal alignment, combined with SIGReg regularization to effectively prevent representation collapse. This approach achieves joint audio-visual encoding under a collapse-free objective for the first time, substantially simplifying model design. The proposed method attains 36.0 mAP on AudioSet, 91.3% accuracy on ESC-50, and 61.1% accuracy on VGGSound, while further supporting zero-shot retrieval capabilities.

Abstract

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Problem

• audio-visual self-supervised learning
• minimalist architecture
• representational collapse
• cross-modal alignment

Innovation

• Audio-Visual Self-Supervised Learning
• LeJEPA
• Modality Dropout
• Early Fusion
• Representation Collapse Prevention