SoundscapeAgent: Agentic Soundscape Construction for Controllable Synthesis and Scalable Audio-Language Supervision

2026-07-23

AI Summary

Current text-to-audio models implicitly handle sound scene construction, lacking explicit control and interpretability over the composition of sound events, their temporal arrangement, and the rendering process. This work proposes the first agent-based soundscape synthesis framework, which leverages a large language model to translate user intent into an editable scene plan. By integrating audio retrieval with on-demand generation, the framework enables controllable multi-event mixed rendering and supports human-in-the-loop interaction and tool selection. It explicitly models the full pipeline of sound scene creation—planning, source selection, layout, and rendering—producing audio that matches state-of-the-art text-to-audio models in both subjective listening quality and objective metrics. Moreover, downstream audio reasoning models trained on its synthetic data significantly outperform baselines trained solely on real-world recordings.

Abstract

We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.

Problem

• controllable audio generation
• soundscape synthesis
• audio-language supervision
• compositional audio
• scene planning

Innovation

• agentic soundscape construction
• controllable audio generation
• audio-language supervision
• LLM-based planning
• human-in-the-loop interaction