neural audio synthesis

Current text-to-audio (T2A) models struggle to achieve precise, fine-grained control over acoustic attributes, hindering personalized audio generation. To address this, we propose the first framework tailored for customized T2A synthesis, leveraging reference-audio-guided diffusion modeling integrated with large language models and dual-modality alignment training—enabling joint modeling of semantic fidelity and target acoustic characteristics (e.g., timbre, rhythm, event structure). Our method extracts and faithfully reproduces speaker- or style-specific acoustic features from only a few reference samples. We introduce two dedicated datasets, including the CTTA benchmark with real-world scene annotations. Experiments demonstrate that our model significantly outperforms state-of-the-art methods on customized generation tasks, achieving superior audio quality, semantic alignment, and acoustic feature fidelity—while maintaining competitive performance on general-purpose T2A benchmarks.

To address the challenges of subjective quality assessment and the inefficiency of purely data-driven models in speech/audio coding, this paper proposes a tightly integrated hybrid neural coding framework that synergistically combines model-driven and data-driven paradigms. Methodologically, it introduces a novel multi-level hybrid architecture that deeply couples psychoacoustic-weighted loss, customized time-frequency domain prediction (TF-Codec/MDCTNet), an LPCNet-based backbone, and a neural post-processing module, trained end-to-end via an autoencoder paradigm. The core contribution lies in systematically bridging the performance gap between classical signal modeling and end-to-end deep learning. Experimental results demonstrate that, at ultra-low bitrates of 1.6–3.2 kbps, the proposed method achieves a P.808 MOS gain of ≥0.5 over baselines, yielding subjective audio quality approaching that of wideband codecs, while increasing computational overhead by less than 15%.

To address the challenges of incomparable cross-task performance and narrow evaluation dimensions in neural audio, music, and speech coding, this paper introduces the first open-source unified training and evaluation platform. Methodologically, it proposes ESPnet-Codec—an integrated codec framework—and VERSA—a standalone evaluation toolkit—supporting mainstream models (e.g., SoundStream, EnCodec, DAC) with discrete quantization, residual vector quantization (RVQ), and multi-scale adversarial training. It enables fully automated assessment across 20 objective audio metrics and seamless integration with six ESPnet downstream tasks. The key contributions are: (1) establishing the first cross-modal neural codec benchmark; (2) significantly improving training efficiency and interoperability for downstream applications (e.g., TTS, music generation); and (3) achieving state-of-the-art performance on both objective metrics and subjective MOS scores, with full reproducibility.

Audio representation learning heavily relies on large-scale real-world recordings, while manual annotation and data augmentation struggle to capture the full diversity of physical acoustics. Method: We propose a synthetic-driven contrastive learning framework that requires no real audio data. It employs differentiable and stochastic sound synthesizers to generate physically consistent synthetic “twin” positive pairs online via causally interpretable parameter perturbations (e.g., timbre, pitch, envelope), thereby constructing high-diversity contrastive tasks. Contribution/Results: We introduce the first positive-pair construction paradigm grounded in causal perturbations of synthesizer parameters; require only a single interpretable hyperparameter and zero real-data storage; and achieve, for the first time, synthetic-data-only models that surpass real-data baselines on ESC-50, UrbanSound8K, and SpeechCommands. This significantly reduces data dependency and storage overhead, establishing a new paradigm for low-resource audio representation learning.

To address data scarcity in low-resource audio classification, this paper proposes a novel data augmentation framework integrating text-to-audio (T2A) diffusion modeling, preference optimization via proximal policy optimization (PPO), and large language model (LLM)-driven iterative caption generation. The method employs PPO to align synthesized audio with target acoustic characteristics, while the LLM dynamically generates and refines semantically diverse, high-quality captions—jointly enhancing acoustic fidelity and semantic richness. Distinct from prior work, this is the first study to holistically unify T2A diffusion, human-feedback-based preference optimization, and LLM-guided iterative captioning. Evaluated across 10 benchmark datasets under four low-resource settings, the framework achieves substantial performance gains (+0.1%–39%) using only a weakly supervised AudioSet-pretrained T2A model, consistently outperforming state-of-the-art baselines.

Training end-to-end text-to-speech (TTS) models with purely synthetic data remains underexplored, particularly regarding feasibility, robustness, and controllability compared to real speech data. Method: This study systematically evaluates FastSpeech 2- and VITS-based TTS models trained exclusively on synthetic speech, conducting ablation experiments by controlling textual richness, speaker diversity, environmental noise level, and speaking style. Evaluation integrates MOS, WER, CMOS, and subjective listening tests. Contribution/Results: To our knowledge, this is the first empirical demonstration that synthetic-data-only training achieves a MOS of 4.12—significantly surpassing the real-data baseline (3.78) at equivalent scale. The synthetic-trained models exhibit 27% higher robustness to accent and noise, and 31% improved cross-speaker generalization similarity. Key findings identify high text/speaker diversity and low environmental noise as primary drivers of robustness, while standard speaking style accelerates convergence. These results establish a theoretically grounded, cost-effective paradigm for controllable, high-quality TTS data curation.

This work addresses the absence of evaluation benchmarks capable of precisely matching real-world audio recordings with their conditionally generated synthetic counterparts. The authors introduce Doppelganger, a benchmark comprising 10,420 real-synthetic sound effect pairs and seven categories of controlled stimuli, and formally define, for the first time, a cross-domain sound matching task spanning real and synthetic audio. Using a contrastive embedding model trained on paired data, the approach achieves 80% source identification accuracy on unseen sound events—substantially outperforming a class-label baseline (61%) and random chance (0.03%). Additional experiments reveal that human listeners misidentify synthetic sounds as real with 29% probability, whereas generator-specific detectors can nearly perfectly discriminate between the two. This study underscores the critical role of paired supervision in learning effective cross-domain audio representations.

This work addresses the ambiguity in labeling resynthesized audio generated by neural audio codecs—a class of models that combine compression and synthesis capabilities—within the context of voice spoofing detection. The study presents the first systematic analysis of this labeling challenge, introducing an extended version of the ASVspoof 5 dataset and proposing multiple annotation strategies tailored to resynthesized audio. A unified evaluation framework is designed to assess the impact of different labeling approaches on anti-spoofing systems, leveraging resynthesis techniques that integrate neural codecs with vocoders. Experimental results demonstrate that the choice of annotation strategy significantly influences detection performance, offering critical insights for future dataset construction and evaluation protocols in audio deepfake detection research.

This work addresses the challenge of unified generation of speech, sound effects, and music—a task requiring high fidelity, end-to-end training, effective contextual modeling, and variable-length synthesis, which existing approaches struggle to balance. The authors propose AudioCALM, a framework that extends autoregressive language modeling to continuous audio latent spaces by integrating flow matching (predicting rectified-flow velocities) with a block-causal AR-Flow attention mechanism, enabling high-quality generation of arbitrary duration. A novel asymmetric multimodal mixture-of-experts architecture (A-MoME) and a unified conditioning interface are introduced: speech utilizes dedicated residual experts, while sound effects and music share a common backbone, incurring no additional inference cost. Experiments demonstrate that AudioCALM matches or surpasses specialized models across all three modalities and significantly outperforms current unified audio generation methods.

To address the substantial storage overhead, low transmission efficiency, and heightened privacy risks inherent in continuous speech representations, this paper proposes Codec2Vec—the first self-supervised speech representation learning framework entirely based on discrete acoustic units from neural audio codecs. Codec2Vec employs a masked discrete unit prediction objective, jointly optimizing multiple targets to learn robust and compact speech representations. On the SUPERB benchmark, it achieves performance comparable to continuous-input models while reducing storage requirements by up to 16.5× and accelerating training by 2.3×. Crucially, its discrete tokenization enables native on-device data anonymization, significantly enhancing privacy preservation and system scalability. The core contribution lies in pioneering the direct modeling of discrete speech codes as the fundamental units for self-supervised learning—establishing a new paradigm for efficient, secure, and lightweight speech representation learning.