UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception

2026-05-29

AI Summary

This work addresses the limitations of existing semantic speech tokenizers, which overly emphasize linguistic abstraction and consequently lack robust perception of general audio, hindering their applicability to non-speech-centric tasks. To overcome this, the authors propose a unified single-codebook semantic tokenizer that introduces Semantic-Acoustic Primitives (SAPs) as structured supervisory signals. The architecture incorporates a content-aware gating mechanism and a shallow acoustic detail recovery module to jointly model linguistic and acoustic information. This approach preserves high-fidelity speech generation while significantly enhancing the model’s capacity to understand and represent general audio. Empirical results demonstrate consistent outperformance over current single-codebook baselines across diverse audio understanding and generation benchmarks, and the tokenizer functions effectively as a unified audio interface for integration with large language models.

Abstract

Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive evaluations show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks, effectively serving as a unified audio interface. We publicly release all our code, including training and inference scripts, together with the model checkpoints at https://github.com/Tencent/Universal_Audio_Tokenizer.

Problem

• semantic speech tokenizers
• acoustic blindness
• general audio perception
• audio representation
• speech-centric tasks

Innovation

• Semantic-Acoustic Primitives
• Semantic-Acoustic Equilibrium
• universal audio tokenizer
• audio-LLM interface
• single-codebook representation