CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits

2026-08-06

AI Summary

Current approaches to controlling the behavior of large language models predominantly rely on single-layer, static interventions, which struggle to achieve consistent and fluent regulation across semantically diverse inputs. This work proposes a multi-layer semantic circuit intervention framework based on sparse autoencoders: by identifying cross-layer semantic pathways through feature co-activation and geometric alignment of decoding directions, it synthesizes dense guidance vectors to enable coordinated multi-point interventions. For the first time, this method achieves precise multi-layer control grounded in geometric alignment, overcoming the limitations of single-point static interventions. It significantly outperforms existing techniques on complex behavioral steering tasks—including toxicity suppression, emotional intensity modulation, sycophancy reduction, and refusal elicitation—while preserving output fluency and semantic coherence.

Abstract

Controlling the behavior of large language models (LLMs) remains a critical challenge for AI alignment. Existing steering methods, such as Contrastive Activation Addition (CAA), typically rely on fixed single-layer interventions derived from aggregate activation differences. These methods impose a single intervention across semantically diverse inputs and often fail to sustain consistent behavioral changes across layers, limiting the effectiveness of the steering. In this work, we introduce CircuitSteer, a novel framework that leverages Sparse Autoencoders (SAEs) to identify and manipulate coherent semantic circuits distributed across multiple layers. By constructing a feature flow circuit based on feature co-activation and the geometric alignment of decoder directions, we isolate the specific multi-layer subcircuits responsible for a target behavior. We then synthesize dense steering vectors from these sparse features and apply multi-point interventions to guide the model's internal semantic trajectory. We evaluate CircuitSteer using contrastive examples across a diverse set of tasks, including toxicity, emotion-intensity, sycophancy, and refusal, spanning two model families. Across all models and datasets, CircuitSteer is the only method to consistently produce fluency-preserving interventions; competing methods either sacrifice text quality or lack coverage, failing entirely on complex behaviors like sycophancy and refusal. These results demonstrate that multi-layer circuit steering, enabled by enforcing geometric alignment among selected features, yields strictly more robust and effective behavioral control than static single-point interventions. Code is available at https://github.com/mehrshad-sdtn/CircuitSteer.

Problem

• AI alignment
• behavioral control
• large language models
• steering methods
• semantic circuits

Innovation

• Sparse Autoencoders
• Multi-Layer Steering
• Geometric Alignment
• Feature Flow Circuit
• Behavioral Control