Sonic Stage: Automatically Generating Interactive Spatial Soundscapes to Facilitate Dialogue Video Comprehension for Blind Viewers

2026-07-22

AI Summary

This study addresses the limitations of traditional audio description in dialogue-dense film and television scenes, where avoidance of overlapping speech often omits critical visual actions and spatial cues, thereby hindering comprehension for blind viewers. To overcome this, the authors propose an interactive spatial soundscape approach that, for the first time, integrates spatialized dialogue, diegetic sound effects, and context-aware interactive audio description to convey visual content synchronously without interfering with spoken dialogue. Moving beyond linear audio description paradigms, this method enables user-driven auditory interaction. Evaluation with twelve blind and low-vision participants demonstrates significant improvements in video comprehension, spatial presence, and narrative engagement, underscoring the innovative potential and broad applicability of immersive audio techniques in accessible media design.

Abstract

Audio description (AD) makes film and television accessible to blind and low-vision (BLV) audiences by narrating characters' actions. However, in scenes with lots of dialogue, AD often omits important actions because it is constrained not to overlap with speech. It is not yet known how to convey characters' actions during dialogue. We present Sonic Stage, a system that transforms dialogue videos into interactive spatial soundscapes, enabling BLV audiences to intuitively understand characters' actions and movements through immersive auditory cues. Sonic Stage conveys essential visual information during dialogue through three auditory techniques: (1) spatialized dialogue to represent spatial layout, (2) diegetic sound to convey character actions, and (3) interactive descriptions to provide context-specific visual details. Evaluation with 12 BLV viewers showed that Sonic Stage significantly improved video comprehension, spatial presence, and narrative engagement. We highlight opportunities for enhancing video accessibility across diverse genres through immersive, interactive audio representations.

Problem

• audio description
• blind viewers
• dialogue video comprehension
• visual accessibility
• spatial soundscapes

Innovation

• spatial audio
• audio description
• interactive soundscapes
• accessibility
• diegetic sound