LILAC: An Idempotent Neural Speech Codec

2026-08-06

AI Summary

This work addresses the lack of idempotence in existing neural speech codecs, which suffer from at least 15% token alteration across successive encode-decode cycles, undermining the reliability of speech processing pipelines. To resolve this, the authors propose LILAC—the first fully convolutional neural speech codec that achieves constructive idempotence, guaranteeing that any valid token remains strictly unchanged after decoding followed by re-encoding. Operating at an ultra-low bitrate of 0.75 kbit/s and a frame rate of 9.375 Hz, LILAC incorporates carefully designed constraints to jointly preserve idempotence and high reconstruction fidelity. It attains UTMOS scores of 4.14 on LibriSpeech and 4.24 on LibriTTS-R, establishing state-of-the-art performance among sub-1 kbit/s neural audio codecs.

Abstract

Neural Audio Codecs are widely adopted in speech generation and editing. However, existing neural audio codecs are not idempotent: across the paper's twelve baseline systems, every configuration tested rewrites, on average, at least 15% of its tokens in a single decode-re-encode pass. This poses a problem for utilizing Neural Audio Codecs as token interfaces in pipelines where re-encoding decoded outputs can occur. We present LILAC, a fully convolutional 24 kHz speech codec at 9.375 Hz and 0.75 kbit/s that is codec idempotent by construction; re-encoding the decoded audio of any valid token stream returns the identical stream. LILAC achieves idempotency while maintaining competitive quality, reaching UTMOS 4.14 and 4.24 on LibriSpeech and LibriTTS-R test sets, comparable to SOTA sub-1 kbit/s Neural Audio Codecs.

Problem

• neural audio codecs
• idempotency
• speech codec
• token interface
• re-encoding

Innovation

• idempotent codec
• neural speech codec
• token stability
• low-bitrate audio coding
• convolutional audio model