Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing

2026-08-05

AI Summary

This work addresses the challenge of speech inpainting and editing, where preserving speaker identity, prosody, temporal structure, and recording conditions is crucial. The authors propose SIEDD, a hierarchical discrete diffusion model based on a layered codec architecture. SIEDD is the first to explicitly integrate hierarchical codec structure into the discrete diffusion process, enabling leakage-free joint training and coarse-to-fine inference through sequential modeling with residual vector quantization (RVQ). The framework further incorporates phoneme-level text guidance, span-localized classifier-free guidance, and duration prediction, supporting both fixed- and variable-length editing. Evaluated on the RealEdit benchmark, SIEDD substantially outperforms existing autoregressive baselines and achieves state-of-the-art performance in both single- and multi-segment inpainting tasks.

Abstract

Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.

Problem

• speech inpainting
• speech editing
• text-guided generation
• context consistency
• missing speech reconstruction

Innovation

• discrete diffusion
• hierarchical codec
• text-guided speech editing
• speech inpainting
• context-preserving generation