StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis

2026-07-22

AI Summary

This work addresses the challenge of simultaneously achieving robustness, low latency, and natural prosody in text-to-speech (TTS) synthesis by proposing a non-autoregressive TTS framework tailored for mobile deployment. The approach introduces sparse temporal embeddings to enable fine-grained control over phoneme duration, pronunciation, and prosody, while integrating a semantic-aware encoder–decoder architecture that supports efficient single-pass decoding. Built upon a lightweight masked generative Transformer with 83 million parameters, the system achieves audio quality, prosodic naturalness, and speaker similarity on par with state-of-the-art models, yet attains a real-time factor of 0.08—significantly reducing inference latency and enhancing robustness.

Abstract

The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-autoregressive (NAR) alternatives, while fast, often sacrifice prosodic naturalness via rigid alignments. This paper introduces StellarTTS, a novel mobile-optimized NAR TTS framework based on a sparse temporal embedding strategy, enabling granular control of phoneme duration, pronunciation, and prosody. Furthermore, we propose a semantic-aware codec that facilitates efficient single-stage decoding. Conditioned on the sparse temporal embedding, our 83M-parameter lightweight masked generative transformer achieves a real-time factor (RTF) of 0.08. Experiments demonstrate that StellarTTS attains lower latency and stronger robustness compared to state-of-the-art TTS systems, while maintaining competitive performance in audio quality, prosodic naturalness, and speaker similarity.

Problem

• text-to-speech
• latency
• robustness
• prosody
• non-autoregressive

Innovation

• sparse temporal embedding
• non-autoregressive TTS
• semantic-aware codec
• low-latency speech synthesis
• masked generative transformer