Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

2026-08-04

AI Summary

This work addresses the challenges in audio-visual instance segmentation caused by overlapping sound sources, which complicate audio-visual correspondence, and temporal asynchrony, which undermines tracking stability. To this end, the authors propose a novel framework featuring an Acoustic-Semantic Projector (ASP) that establishes hierarchical cross-modal mappings from semantics to spatial locations to disentangle mixed audio signals, and an Asynchronous Dynamics Modulator (ADM) that leverages audio cues to modulate Mamba-based state transitions dynamically. This enables pixel-level accurate segmentation and robust tracking of sounding objects. Notably, the method introduces a state-aware mechanism for the first time in this domain, achieving a new state-of-the-art performance of 48.54 mAP on the AVISeg benchmark—surpassing the previous best approach by 7.8%.

Abstract

Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8\%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.

Problem

• audio-visual instance segmentation
• overlapping sound sources
• temporal misalignment
• asynchronous dynamics

Innovation

• Audio-Visual Instance Segmentation
• Acoustic-Semantic Projector
• Asynchronous Dynamics Modulator
• Mamba
• Stateful Listening