LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

2026-09-25

AI Summary

This study addresses the issue that activation steering tends to entangle irrelevant attributes and compromise the general capabilities of language models. To this end, we propose a training-free, precise local inference-time intervention method. Specifically, our approach leverages the unembedding matrix to localize the output vocabulary subspace and employs a sparse attention head selection mechanism to achieve targeted intervention. Furthermore, geometric constraints are introduced to restrict the steering scope, modifying fewer than 6% of the parameters. Experimental results demonstrate that this paradigm significantly outperforms existing baselines on tasks such as toxicity mitigation, achieving precise control while effectively preserving the model's original general-purpose capabilities.

Abstract

Activation steering is a powerful training-free paradigm for controlling large language models at inference time. However, standard approaches estimate a per-layer steering direction from contrastive data and apply it on the layer's entire representation space, which may couple the intervention to off-target properties present in the contrastive data and degrade unrelated capabilities. To mitigate this issue, we introduce LocUS (Localized Unembedding Steering), a method which grounds activation steering to the model's own output vocabulary subspace. By identifying a property-specific linear subspace within the unembedding matrix, LocUS enforces a geometric constraint that restricts the steering transformation to a specific subspace and at the same time localizes its application to a sparse subset of attention heads. Extensive evaluations across three model families on toxicity mitigation, sentiment redirection and sycophancy suppression show that LocUS matches or outperforms state-of-the-art baselines while intervening on under 6% of parameters and better preserving general capability.

Problem

• Activation Steering
• Large Language Models
• Off-target Coupling
• Capability Degradation

Innovation

• Activation Steering
• Subspace Projection
• Head Selection
• Unembedding Matrix
• Training-free Intervention