Curveball Steering: The Right Direction To Steer Isn't Always Linear

2026-03-10

AI Summary

This work addresses a critical limitation in existing activation intervention methods for large language models, which rely on the assumption of linear representations and struggle to stably control model behavior in nonlinear activation spaces. The study reveals, for the first time, that the activation space of large language models exhibits significant geometric nonlinear distortion. To overcome this, the authors propose Curveball steering, a novel nonlinear intervention method based on polynomial kernel PCA. By mapping activations into a feature space that better reflects their intrinsic geometric structure, Curveball steering enables more accurate behavioral control. The degree of distortion is quantified via the ratio between geodesic and Euclidean distances. Experimental results demonstrate that, particularly in high-distortion regimes, Curveball steering substantially outperforms conventional linear approaches, achieving more consistent and effective intervention and thereby transcending the constraints of the linear intervention paradigm.

Technology Category

• Machine Learning: Active Learning
• Natural Language Processing: (Large) Language Models
• Intelligent Robots: Behavior Learning & Control

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Activation steering is a widely used approach for controlling large language model (LLM) behavior by intervening on internal representations. Existing methods largely rely on the Linear Representation Hypothesis, assuming behavioral attributes can be manipulated using global linear directions. In practice, however, such linear interventions often behave inconsistently. We question this assumption by analyzing the intrinsic geometry of LLM activation spaces. Measuring geometric distortion via the ratio of geodesic to Euclidean distances, we observe substantial and concept-dependent distortions, indicating that activation spaces are not well-approximated by a globally linear geometry. Motivated by this, we propose "Curveball steering", a nonlinear steering method based on polynomial kernel PCA that performs interventions in a feature space, better respecting the learned activation geometry. Curveball steering consistently outperforms linear PCA-based steering, particularly in regimes exhibiting strong geometric distortion, suggesting that geometry-aware, nonlinear steering provides a principled alternative to global, linear interventions.

Problem

• activation steering
• linear representation hypothesis
• geometric distortion
• nonlinear intervention
• LLM activation space

Innovation

• nonlinear steering
• activation geometry
• kernel PCA
• representation intervention
• geodesic distortion