When is Your LLM Steerable?

2026-06-09

AI Summary

Existing activation steering methods suffer from inconsistent efficacy and rely on full-generation evaluations, which are computationally expensive. This work proposes a novel approach that analyzes the multi-layer, multi-step hidden state dynamics during the early stages of model generation to extract structured features predictive of steering outcomes. Leveraging these features, we construct a steerability predictor based on gradient-boosted decision trees (GBDT). We introduce ASTEER, the first large-scale benchmark for systematic evaluation of activation steering, demonstrating that early hidden states contain reliable signals for predicting intervention success or failure. The proposed method achieves a macro F1 score of approximately 0.7 on unseen concepts and enables highly efficient selection of intervention strength with minimal decoding overhead, closely approximating optimal search performance.

Abstract

Activation steering offers a lightweight approach to control language models' behavior at inference time, but whether it succeeds or fails heavily depends on the prompt, concept, model, and steering configuration. Finding the regime and boundaries of successful steering typically requires expensive grid searches and post-hoc evaluation of full autoregressive rollouts. In this work, we investigate whether steerability can be predicted from the model's internal states at the beginning of the generation process, e.g., after generating the first few tokens, and how to leverage such a predictor to improve steering success rate. To this end, we first introduce ASTEER, a testbed including 1.4M steered generations, spanning 150 concepts with each steering success/failure labeled. Leveraging this testbed, we analyze the model's early decoding dynamics by extracting features that compare hidden states before and after steering across layers and initial decoding steps. These features help us understand how steering's effects propagate along layers and token positions, which provide key information for steerability prediction. We then train a Gradient Boosting Decision Trees (GBDT) classifier on these features to predict whether an intervention will under-steer, succeed, or over-steer without requiring full rollout. Our predictor achieves around 0.7 macro-F1 score on unseen concepts, demonstrating that early hidden states encode substantial, structured information about eventual steering efficacy. We further leverage this steerability predictor as guidance for steering strength searching, achieving near-optimal performance with a small fraction of decoding cost.

Problem

• steerability
• activation steering
• language models
• inference-time control
• steering prediction

Innovation

• activation steering
• steerability prediction
• early hidden states
• ASTEER
• GBDT classifier