Rewriting or Reweighting? A Geometric Account in Language Models

2026-08-03

AI Summary

This work investigates the mechanistic nature of behavioral changes in language model post-training—whether such changes arise from rewriting, creating new mechanisms, or merely reweighting existing ones. To this end, the authors propose Behavior Manifold Analysis, a method that constructs low-dimensional local charts in both activation space (ACT) and neuron output contribution space (NOC) to trace how supervised fine-tuning (SFT) and reward optimization reshape behavioral geometry. For the first time from a geometric perspective, the study reveals that SFT substantially reconstructs the behavior manifold, whereas reward optimization largely preserves its underlying structure and primarily adjusts output weights. Experiments across multiple model architectures demonstrate the high compressibility of these charts and their partial cross-model alignment, highlighting the method’s novel utility for mechanistic interpretability and cross-architecture comparison.

Abstract

Post-training can substantially alter language-model behavior, yet aggregate behavior rates do not reveal whether training removes an existing mechanism, creates a new one, or changes how an inherited mechanism is used. We study this question through two mechanistically distinct failures, repetition as a decoding-attractor pathology and sycophancy as a preference-related alignment failure. We introduce behavioral manifold analysis, which isolates behavior-specific geometry by selecting sparse behavior-associated coordinates and lifting them into low-dimensional local charts. We construct these charts in two complementary spaces. ACT captures runtime activation states, while NOC quantifies how strongly the model routes functional information flow through the shared behavior-associated subspace. Across multiple model families, the resulting charts are highly compressed and partially alignable across architectures. Contribution-space charts expose a more architecture-robust shared core, whereas activation-space charts retain stronger family-specific structure. Tracking these charts through controlled post-training reveals a consistent asymmetry. Supervised fine-tuning substantially alters the inherited behavioral geometry, whereas reward optimization changes behavior while largely preserving the underlying chart. This geometric perspective provides a unified framework for understanding the mechanistic distinction between the two objectives. SFT tends to rewrite behavioral geometry, whereas reward optimization primarily reweights it. Code is available at https://github.com/ronglingze/Manifold-Analysis

Problem

• post-training
• behavioral mechanism
• language models
• supervised fine-tuning
• reward optimization

Innovation

• behavioral manifold analysis
• activation space
• contribution space
• post-training dynamics
• geometric perspective