Technology Category
Machine Learning: Deep Generative Models & AutoencodersNatural Language Processing: GenerationComputer Vision: Large Vision Models
Taming Data and Transformers for Scalable Audio Generation
🤖 AI Summary
To address three key bottlenecks in environmental sound generation—data scarcity, low-quality captions, and limited model scalability—we propose a data-model co-scaling paradigm. First, we construct AutoReCap-XL, the first ultra-large-scale, high-quality audio-text dataset comprising 47 million segments. Second, we design AutoCap, a high-fidelity automatic captioning model integrating Q-Former with audio metadata and incorporating synthetic caption distillation. Third, we introduce GenAu, a scalable Transformer architecture (1.25B parameters) tailored for long-duration audio generation, enhanced via contrastive learning and architectural scaling optimization. Experiments show AutoCap achieves a CIDEr score of 83.2 (+3.2% absolute improvement), while GenAu outperforms prior models significantly (FAD ↓4.7%, IS ↑11.1%, CLAP ↑13.5%). All code, models, and datasets are publicly released.
Technology Category
Application Category
📝 Abstract
The scalability of ambient sound generators is hindered by data scarcity, insufficient caption quality, and limited scalability in model architecture. This work addresses these challenges by advancing both data and model scaling. First, we propose an efficient and scalable dataset collection pipeline tailored for ambient audio generation, resulting in AutoReCap-XL, the largest ambient audio-text dataset with over 47 million clips. To provide high-quality textual annotations, we propose AutoCap, a high-quality automatic audio captioning model. By adopting a Q-Former module and leveraging audio metadata, AutoCap substantially enhances caption quality, reaching a CIDEr score of $83.2$, a $3.2%$ improvement over previous captioning models. Finally, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. We demonstrate its benefits from data scaling with synthetic captions as well as model size scaling. When compared to baseline audio generators trained at similar size and data scale, GenAu obtains significant improvements of $4.7%$ in FAD score, $11.1%$ in IS, and $13.5%$ in CLAP score. Our code, model checkpoints, and dataset are publicly available.
Problem
Research questions and friction points this paper is trying to address.
Addressing data scarcity in ambient audio generation
Improving caption quality for audio-text datasets
Enhancing scalability of transformer-based audio models
Innovation
Methods, ideas, or system contributions that make the work stand out.
Efficient scalable dataset pipeline for audio
High-quality automatic captioning model AutoCap
Scalable transformer-based architecture GenAu
🔎 Similar Papers
No similar papers found.
