MeloCodec: Harnessing Melodic Priors for High-Fidelity Singing Voice Representation

2026-08-03

AI Summary

Existing neural audio codecs struggle to effectively leverage explicit melodic priors in singing voice synthesis, often resulting in insufficient pitch fidelity and degraded high-frequency details. To address this limitation, this work proposes MeloCodec, a novel framework that introduces a Tokenize-then-Fuse paradigm: melodic information is first discretized and structurally fixed before being fused with audio features. Combined with a two-stage training strategy, this approach robustly mitigates codebook collapse and enables stable, efficient integration of melodic priors. Experimental results demonstrate that MeloCodec significantly enhances pitch consistency while preserving timbral quality, supports high-fidelity and controllable pitch editing, and outperforms current state-of-the-art baselines.

Abstract

Neural audio codecs serve as fundamental tokenizers for LLM-based audio generation. While semantic priors are widely exploited to enhance linguistic intelligibility, the integration of explicit acoustic priors remains underexplored, limiting synthesis fidelity in frequency-sensitive domains. To address this gap, we introduce MeloCodec, a novel framework designed to effectively incorporate melodic priors, a critical form of acoustic information for singing. To address the optimization instability typically caused by the direct fusion of such explicit priors, we propose a Tokenize-then-Fuse paradigm that pre-trains a discrete melodic branch to lock in structures before feature fusion. To robustly realize this paradigm, we further propose a two-stage training strategy that prevents codebook collapse and ensures stable convergence. Experiments show that MeloCodec outperforms baselines in singing voice representation, improving pitch consistency and enabling controllable pitch manipulation with minimal timbre degradation.

Problem

• neural audio codecs
• acoustic priors
• melodic priors
• singing voice representation
• synthesis fidelity

Innovation

• MeloCodec
• melodic priors
• neural audio codec
• Tokenize-then-Fuse
• two-stage training