SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

2026-05-29

AI Summary

This work addresses the limited out-of-domain generalization and ineffective speech–language representation fusion in existing speech-aware large language models. We propose a lightweight adaptation method that abandons conventional guidance mechanisms relying on contrastive activation differences and instead introduces, for the first time, direct supervised optimization of layer-wise guiding activation vectors. Our analysis reveals that steering the speech encoder—particularly its deeper layers—is more effective than adapting the LLM backbone. The proposed approach substantially outperforms zero-shot inference and speech-in-context learning baselines on challenging benchmarks involving child speech, multilingual inputs, and Mandarin–English code-switching, achieving up to a 46.8% relative performance improvement.

Abstract

Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself.

Problem

• speech-aware LLM
• out-of-domain generalization
• ASR
• representation alignment

Innovation

• Learned Steering Vectors
• Speech-Aware LLM Adaptation
• Supervised Activation Steering
• Encoder Steering
• Acoustic-Phonetic Alignment