Look Before You Steer: Geometry Predicts SAE Feature Steerability

2026-09-19

AI Summary

研究通过分析SAE的几何特性来预测特征的可操控性，从而减少对每个特征进行系数调整的需求，提供了一种在部署前筛选可控特征的方法。

Abstract

Steering with SAE features requires per-feature coefficient tuning, which currently demands intervention sweeps. We ask whether properties of the SAE itself, computable before any forward pass, predict which features will be cheap or expensive to steer. We show that variation in SAE feature steerability is partially predicted by decoder-space geometry: neighbor density and maximum cosine similarity to nearby decoder directions, both computable from the SAE weight matrix before any intervention, rank features by how much steering they require for a fixed behavioral effect ($ρ$ up to $-0.546$, $p < 10^{-6}$, AUROC 0.610-0.822 across conditions; the signal is rank-based, consistent with grid discreteness). This geometry-steerability relationship replicates across two Gemma-2 model scales (2B and 9B), two SAE widths (16K and 65K), and is detectable cross-architecturally on Llama-3.1-8B-Instruct ($ρ= -0.266$, $n = 300$). On Qwen3-8B with BatchTopK SAEs, geometry predicts whether a feature is steerable at all but not the continuous ordering among responsive features, revealing a boundary condition tied to SAE training regime. The signal weakens at deep proportional layer depth in both models, where the cost of steering exceeds our intervention budget, a consistent depth boundary. These results provide preliminary evidence that pre-steering geometry can partially inform coefficient selection, offering a path toward screening features for controllability before deployment.

Problem

• SAE
• feature steerability
• coefficient tuning
• decoder-space geometry
• neighbor density

Innovation

• Geometry Predicts SAE Feature Steerability
• Decoder-Space Geometry
• Coefficient Tuning
• Pre-Steering Prediction
• Cross-Architectural Replication