DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos

2026-04-03

AI Summary

This work addresses the prevalent absence of spatial audio in existing 360-degree videos and the challenge of modeling dynamic sound sources alongside complex acoustic phenomena such as occlusion, reflection, and reverberation. To this end, the authors propose a novel approach that integrates dynamic scene reconstruction with a conditional diffusion model, uniquely incorporating geometric and material information derived from 3D Gaussian splatting into the audio generation pipeline. This integration enables joint modeling of moving sound sources and their interactions with the surrounding acoustic environment. Experimental results on the newly introduced M2G-360 dataset demonstrate that the proposed method significantly outperforms current state-of-the-art techniques in terms of spatial accuracy, acoustic fidelity, and overall immersion.

Technology Category

Computer Vision: 3D Computer VisionMachine Learning: Deep Generative Models & AutoencodersKnowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGUser Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Spatial audio is crucial for immersive 360-degree video experiences, yet most 360-degree videos lack it due to the difficulty of capturing spatial audio during recording. Automatically generating spatial audio such as first-order ambisonics (FOA) from video therefore remains an important but challenging problem. In complex scenes, sound perception depends not only on sound source locations but also on scene geometry, materials, and dynamic interactions with the environment. However, existing approaches only rely on visual cues and fail to model dynamic sources and acoustic effects such as occlusion, reflections, and reverberation. To address these challenges, we propose DynFOA, a generative framework that synthesizes FOA from 360-degree videos by integrating dynamic scene reconstruction with conditional diffusion modeling. DynFOA analyzes the input video to detect and localize dynamic sound sources, estimate depth and semantics, and reconstruct scene geometry and materials using 3D Gaussian Splatting (3DGS). The reconstructed scene representation provides physically grounded features that capture acoustic interactions between sources, environment, and listener viewpoint. Conditioned on these features, a diffusion model generates spatial audio consistent with the scene dynamics and acoustic context. We introduce M2G-360, a dataset of 600 real-world clips divided into MoveSources, Multi-Source, and Geometry subsets for evaluating robustness under diverse conditions. Experiments show that DynFOA consistently outperforms existing methods in spatial accuracy, acoustic fidelity, distribution matching, and perceived immersive experience.

Problem

• spatial audio
• first-order ambisonics
• 360-degree video
• acoustic effects
• dynamic sound sources

Innovation

• Conditional Diffusion
• First-Order Ambisonics
• 3D Gaussian Splatting
• Dynamic Sound Sources
• Spatial Audio Synthesis