Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

2026-08-05

AI Summary

This work addresses the uneven performance of multilingual large language models across languages and the reliance of existing adaptation methods on parameter updates and extensive multilingual data. The authors propose a training-free, inference-stage steering approach that leverages pretrained sparse autoencoders (SAEs) to identify language-relevant features. By contrasting activations from parallel sentence pairs, the method isolates critical features and decodes them into steering signals injected into the model’s hidden states. This is the first approach to integrate SAEs at inference time for data-efficient multilingual enhancement, eliminating the need for fine-tuning or large-scale multilingual corpora. Evaluated on Gemma-2-2B-it, the method achieves gains of 10.9, 5.3, and 1.9 percentage points on XCOPA, XNLI, and MGSM benchmarks, respectively.

Abstract

Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show average accuracy improvements of 10.9 percentage points on XCOPA, 5.3 points on XNLI, and 1.9 points on MGSM.

Problem

• multilingual large language models
• performance disparity
• language adaptation
• inference-time steering
• parameter-efficient adaptation

Innovation

• sparse autoencoder
• inference-time steering
• multilingual LLM
• feature strengthening
• parameter-free adaptation