Equivalence of Context and Parameter Updates in Modern Transformer Blocks

2025-11-21

AI Summary

This work investigates the equivalence between context updates and parameter updates in Transformers. We propose a low-rank (rank-1) weight patching mechanism that implicitly encodes contextual influence as dynamic, input-dependent corrections to MLP layer weights. We establish, for the first time, a formal controllability-theoretic framework—proving that such context-to-parameter mapping is exactly realizable in MLP blocks satisfying mild structural conditions. Our formulation rigorously accommodates modern architectural components, including RMSNorm, gating mechanisms, and pre-/post-norm configurations, thereby unifying the explanation of context-driven parameter adaptation across Gemma, MoE, and deep stacked architectures. Experiments confirm exact equivalence of context effects under this mechanism in both Gemma-style modules and multi-layer models, with broad applicability across mainstream LLM architectures. This work provides a novel theoretical and mechanistic paradigm for understanding context awareness in large language models.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights. This work extends that foundational theory to the diverse architectures of modern Large Language Models. We first demonstrate a precise, analytical solution for a Gemma-style transformer block, proving that the entire effect of a context can be perfectly mapped to rank-1 patches on its MLP weight matrices and a patch to the RMSNorm scale. We then generalize this result, providing a constructive proof and algorithm for multi-layer models. To unify these findings, we introduce a general framework centered on two core properties: input controllability and output controllability. We prove that a perfect implicit weight patch is possible for any MLP block where the inner function is input-controllable and the outer function is output-controllable. This provides a simpler and more powerful lens for understanding how transformer models transmute prompts into effective weights. This setup generalizes to a wide range of modern LLM architectures including gating, pre-/post-norm, mixture of experts and sequential/parallel transformer blocks.

Problem

• Extends context-parameter equivalence theory to modern LLM architectures
• Proves context effects map to rank-1 patches in MLP weights
• Establishes controllability framework for understanding prompt-to-weight transformation

Innovation

• Context represented as rank-1 MLP weight patches
• Analytical solution for Gemma-style transformer blocks
• General framework with input and output controllability