Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data

2024-10-02

arXiv.org

AI Summary

To address data scarcity in low-resource audio classification, this paper proposes a novel data augmentation framework integrating text-to-audio (T2A) diffusion modeling, preference optimization via proximal policy optimization (PPO), and large language model (LLM)-driven iterative caption generation. The method employs PPO to align synthesized audio with target acoustic characteristics, while the LLM dynamically generates and refines semantically diverse, high-quality captions—jointly enhancing acoustic fidelity and semantic richness. Distinct from prior work, this is the first study to holistically unify T2A diffusion, human-feedback-based preference optimization, and LLM-guided iterative captioning. Evaluated across 10 benchmark datasets under four low-resource settings, the framework achieves substantial performance gains (+0.1%–39%) using only a weakly supervised AudioSet-pretrained T2A model, consistently outperforming state-of-the-art baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We present Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data. Traditional data augmentation techniques, which apply artificial transformations (e.g., adding random noise or masking segments), struggle to create data that captures the true diversity present in real-world audios. To address this shortcoming, we propose to augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmentations is challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have sufficient compositional diversity. To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference optimization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models to (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used to prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captioned AudioSet.

Problem

• Improving audio classification with limited labeled data
• Generating synthetic audio that matches real-world diversity
• Ensuring acoustic consistency and diversity in synthetic data

Innovation

• Uses text-to-audio diffusion models for data augmentation
• Aligns T2A model outputs with small-scale datasets
• Generates diverse captions using Large Language Models