Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

2026-06-08

AI Summary

This work addresses the challenge of effectively integrating continuous speech into frozen large language models (LLMs) without fine-tuning while preserving paralinguistic information. The authors propose Convex Gate (C-Gate), a mechanism that represents each speech frame as a convex combination of LLM word embeddings, thereby constraining its representation to lie within the pre-trained embedding manifold. By leveraging geometric structure rather than discrete tokens as the core interface between speech and the LLM, C-Gate unifies compatibility with the frozen model and expressive continuous representation through convex hull constraints. The approach is further enhanced with causal intervention analysis and multimodal fusion strategies. Experiments demonstrate a 48.7% relative reduction in word error rate on LibriSpeech and achieve emotion recognition performance on par with or surpassing dedicated single-task models.

Abstract

Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a frozen LLM remains challenging. Existing speech-to-LLM interfaces typically operate at two extremes: either enforcing near-discrete token alignment, which benefits transcription but loses paralinguistic information, or learning unconstrained continuous representations, which can drift away from the LLM's input space and degrade autoregressive decoding. In this work, we propose Convex Gate (C-Gate), a speech-to-LLM bridge that constrains all speech representations to lie within the LLM's input embedding manifold with an architectural convex-hull constraint. Concretely, each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. Across automatic speech recognition (ASR) and emotion recognition, C-Gate achieves strong joint performance, improving LibriSpeech WER by up to 48.7% relative while matching or exceeding single-task emotion accuracy. Beyond performance, our analysis reveals a key insight: information is not carried by discrete token identities, but by time-resolved trajectories in the embedding space. Causal interventions confirm that both the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. These results suggest that geometry, rather than token discreteness, is the fundamental design factor in speech-to-LLM interfaces, and provide a controlled regime for studying multimodal integration in frozen LLMs. We release the checkpoint, per-sample outputs, mechanism dumps, and intervention suite for replication.

Problem

• speech-to-LLM integration
• frozen LLM
• continuous speech representation
• embedding manifold compatibility
• paralinguistic information

Innovation

• Convex Gate
• embedding manifold
• speech-to-LLM interface
• continuous representation
• information bottleneck