Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

2026-06-22

arXiv.org

AI Summary

This study addresses the absence of pixel-level annotations and the loss of spatial details in weakly supervised audio-visual sound source localization by proposing the LAIP framework. This method exploits the spatial information embedded within intermediate visual tokens of large-scale retrieval models, leveraging audio guidance to restore fine-grained localization capabilities. Central to this approach is the Audio-informed Spatial Pooling (AiSP) mechanism, which integrates frame-level alignment queries with a lightweight network to replace standard global aggregation modules, thereby effectively preserving spatial structures. Experimental evaluations demonstrate that LAIP achieves state-of-the-art performance on both the AVSBench and AVATAR datasets. Notably, the CIoU metric improves substantially from 13.21 to 26.22, nearly doubling the localization performance compared to existing methods.

Abstract

Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information. To exploit this, we introduce LAIP (\emph{Localization via Audio-Informed Pooling}), a framework that employs a lightweight \emph{Audio-informed Spatial Pooling} (AiSP) to replace the standard global aggregation module. By using frame-aligned audio to query intermediate visual tokens, LAIP recovers localized spatial information that is otherwise discarded by the frozen retrieval pipeline. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter. These findings prove that accurate localization does not need to be learned from scratch; instead, it can be unlocked from existing retrieval representations, providing a unified path for both retrieval and localization tasks.

Problem

• Audio-Visual Sound Source Localization
• Weak Supervision
• Spatial Grounding
• Audio-Visual Retrieval

Innovation

• Audio-Visual Sound Source Localization
• Weak Supervision
• Audio-Informed Spatial Pooling
• Intermediate Visual Tokens
• Large Audio-Visual Retrieval Models