Scaling Analysis of Interleaved Speech-Text Language Models

2025-04-03

AI Summary

This work investigates whether speech-language models (SLMs) trained on interleaved speech-text sequences exhibit superior scaling efficiency compared to textless speech-only models. To this end, the authors conduct systematic scaling experiments across dozens of configurations, leveraging pre-trained text language models (TextLMs) for initialization, interleaved speech-text sequence modeling, synthetic data augmentation, and cross-family TextLM transfer analysis. Their study—first to characterize the scaling dynamics of interleaved SLMs—reveals that scaling model size yields greater gains than increasing training steps; critically, synthetic data quality and the choice of TextLM family are decisive factors for performance leaps. Empirically, the proposed approach achieves state-of-the-art results on speech semantic understanding tasks while substantially reducing computational cost and dependence on real speech data. The model, inference samples, and datasets are publicly released.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. They predict that SLMs require much more compute and data compared to text, leading some to question the feasibility of training high-quality SLMs. However, modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question - Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the scaling-dynamics are significantly different than textless-SLMs, suggesting one should allocate notably more of the compute budget for increasing model size over training tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. Results suggest, that our scaled up model achieves comparable performance with leading models on speech semantic metrics while using less compute and data than other approaches. We open source models, samples, and data - https://pages.cs.huji.ac.il/adiyoss-lab/sims.

Problem

• Do interleaved SLMs scale more efficiently than textless-SLMs?
• How do scaling-dynamics differ between interleaved and textless-SLMs?
• What role do synthetic data and TextLM families play?

Innovation

• Interleaved SLMs scale more efficiently
• Allocate compute for model size
• Synthetic data unlocks SLM potential