🤖 AI Summary
This work proposes a unified framework based on continuous depth-token field theory to systematically model the effects of intervention operations—such as activation patching—on Transformer behavior. Treating the residual stream as a depth-token field, patching interventions are formalized as localized source insertions. By integrating first-order sensitivity analysis, empirical Green’s functions, and adjoint variational methods, the framework characterizes the propagation mechanisms of intervention-induced responses. It is the first to describe patching experiments using the language of field theory, revealing anisotropic response patterns across both depth and token dimensions and accurately predicting local linear responses. Empirical validation demonstrates the method’s effectiveness in prompt-induced behavioral transfer, offering a novel paradigm for interpretability analysis in deep learning models.
📝 Abstract
Mechanistic interpretability often uses activation patching, causal tracing, path patching, and steering directions to reveal behaviorally meaningful directions in Transformer activation space. This paper develops a field-theoretic framework for organizing and predicting such interventions. Treating the residual stream as a depth-token field, we formulate patching as localized source insertion, patch effects as sensitivity-field predictions, downstream propagation as empirical Green-function response, and patch selection as an adjoint variational problem. Empirically, we test the forward response theory in GPT-2-style autoregressive Transformers by applying localized residual-field interventions and observing the induced residual-field differences and logit-difference responses. We identify a bounded local linear regime; predict patch effects from first-order sensitivities across residual sites; measure structured anisotropic propagation across depth and token position; construct response descriptions from high-sensitivity sites and sliced Green operators; and show that prompt-induced residual displacements can transfer answer behavior. These results establish response objects, namely sensitivities, propagated fields, and Green-operator slices, as a practical language for organizing patching experiments and as the forward mathematical basis for formulating patch-site inference and cross-scale transfer.formulated.