InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation

2025-02-28

AI Summary

To address the challenge of jointly ensuring coherence, fine-grained detail reconstruction, and training efficiency in high-fidelity, long-duration (up to 8 minutes), high-sample-rate (48 kHz) music generation, this paper introduces InspireMusic-1.5B-Long. Methodologically, it is the first to unify a Qwen-2.5 autoregressive Transformer with a super-resolution flow-matching model for controllable text-to-audio generation. A single-codebook, semantically rich audio tokenizer is designed to substantially reduce computational overhead. Furthermore, acoustic encoder-decoder architecture and end-to-end joint optimization are employed to preserve long-range structural consistency and high-frequency fidelity. Objective and subjective evaluations demonstrate that InspireMusic-1.5B-Long achieves audio quality and temporal coherence comparable to MusicGen and Stable Audio 2.0, while reducing training cost by approximately 35%. This work establishes a novel paradigm for efficient, high-quality, long-form music synthesis.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce InspireMusic, a framework integrated super resolution and large language model for high-fidelity long-form music generation. A unified framework generates high-fidelity music, songs, and audio, which incorporates an autoregressive transformer with a super-resolution flow-matching model. This framework enables the controllable generation of high-fidelity long-form music at a higher sampling rate from both text and audio prompts. Our model differs from previous approaches, as we utilize an audio tokenizer with one codebook that contains richer semantic information, thereby reducing training costs and enhancing efficiency. This combination enables us to achieve high-quality audio generation with long-form coherence of up to $8$ minutes. Then, an autoregressive transformer model based on Qwen 2.5 predicts audio tokens. Next, we employ a super-resolution flow-matching model to generate high-sampling rate audio with fine-grained details learned from an acoustic codec model. Comprehensive experiments show that the InspireMusic-1.5B-Long model has a comparable performance to recent top-tier open-source systems, including MusicGen and Stable Audio 2.0, on subjective and objective evaluations. The code and pre-trained models are released at https://github.com/FunAudioLLM/InspireMusic.

Problem

• High-fidelity long-form music generation from text and audio prompts.
• Integration of super-resolution and large language models for music synthesis.
• Efficient training with a single audio tokenizer for rich semantic information.

Innovation

• Combines super-resolution and large language models
• Uses single codebook audio tokenizer for efficiency
• Generates high-fidelity music up to 8 minutes