Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

2026-07-26

AI Summary

This work proposes an efficient on-device audio synthesis architecture that achieves high-quality, low-latency streaming generation under stringent compute and memory constraints. The method transforms semantic audio tokens into residual vector quantized (RVQ) representations and introduces a shared deep decoder with DiT-style stage conditioning to jointly generate all RVQ levels, replacing conventional multi-decoder designs. To maintain constant memory complexity, it incorporates causal sliding-window attention along with fixed-window key-value caching. Implemented on the Apple Matrix Coprocessor, the system attains approximately 10 ms per generation step (16× real-time), peaks at only 21 MB of memory usage, and supports continuous synthesis up to 320 seconds. Subjective evaluations show a mean opinion score (MOS) improvement of +0.28 overall and +0.42 for conversational speech.

Abstract

Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.

Problem

• memory efficiency
• on-device audio synthesis
• temporal-depth decoupling
• real-time speech generation
• constant memory complexity

Innovation

• Decoupled Temporal-Depth Processing
• Diffusion Transformer (DiT)
• Residual Vector Quantization (RVQ)
• Constant Memory Complexity
• On-Device Audio Synthesis