Feed-Forward Steering in Transformer Residual Dynamics

2026-08-03

AI Summary

This work investigates the mechanistic role of feedforward networks (FFNs) in the residual dynamics of Transformers. Modeling the FFN as a local tangential vector field acting on each token within the residual space, we extend existing attention-only dynamical theories. Through a spherical particle dynamics model, tangential/radial decomposition, commutator defect analysis, and targeted interventions, we demonstrate for the first time that FFNs predominantly govern the tangential evolution of residual trajectories. We further propose the commutator defect as a criterion for module parallelizability. Evaluated across models from GPT-2 to Llama-3-8B, our framework substantially improves one-step angular prediction accuracy; retaining only the tangential component of the FFN suffices to preserve both performance and output diversity; moreover, layers with low commutator defect admit efficient parallelization.

Abstract

Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FFN) term as a local steering field acting on each token state. The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space, that critical residual directions correspond to nonlinear projective equilibria, and that a commutator defect determines when a finite attention--FFN block can be accurately approximated by a parallel, additive flow. Across GPT-2, Pythia, Mistral, and Llama models, the extended theory improves one-step angular prediction relative to an attention-only baseline, with the contribution of the FFN increasing from GPT-2 to Llama-3-8B. Intervention experiments show that retaining only the tangential FFN component preserves most model quality, whereas retaining only the radial component causes performance to collapse. The tangential component also preserves output diversity under aggregation pressure. As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss, whereas layers with large defects degrade rapidly. These findings support the interpretation of FFN layers as directional steering fields that shape Transformer residual geometry and govern the feasibility of block-level interventions.

Problem

• Transformer residual dynamics
• feed-forward network
• attention mechanism
• residual direction
• dynamical theory

Innovation

• feed-forward steering
• residual dynamics
• tangential FFN component
• commutator defect
• projective equilibria