Non-linear Interventions on Large Language Models

2026-05-14

AI Summary

This work addresses a key limitation in existing large language model (LLM) intervention methods, which predominantly rely on linear assumptions and thus struggle to manipulate implicit features encoded along nonlinear manifolds. To overcome this, the paper introduces a general-purpose nonlinear intervention framework that enables precise control over implicit representations—even in the absence of explicit output signals—through a novel learning mechanism. By integrating nonlinear intervention modeling, implicit feature learning, and refusal bypass control, the proposed approach transcends the constraints of linear representational assumptions. Empirical results demonstrate that the method significantly outperforms linear baselines on refusal bypass tasks, achieving more accurate and reliable steering of model behavior.

Abstract

Intervention is one of the most representative and widely used methods for understanding the internal representations of large language models (LLMs). However, existing intervention methods are confined to linear interventions grounded in the Linear Representation Hypothesis, leaving features encoded along non-linear manifolds beyond their reach. In this work, we introduce a general formulation of intervention that extends naturally to non-linearly represented features, together with a learning procedure that further enables intervention on implicit features lacking a direct output signature. We validate our framework on refusal bypass steering, where it steers the model more precisely than linear baselines by intervening on a non-linear feature governing refusal.

Problem

• non-linear intervention
• large language models
• internal representations
• Linear Representation Hypothesis
• implicit features

Innovation

• non-linear intervention
• large language models
• representation learning
• implicit features
• refusal bypass