🤖 AI Summary
This work investigates whether the grokking phenomenon observed in Transformers on modular addition tasks stems from architectural degrees of freedom and proposes structural interventions to eliminate the memorization phase. By introducing spherical embeddings (L2-normalized residual streams), a fixed-temperature decoder, and uniform attention mechanisms, the study imposes geometric inductive biases to actively shape training dynamics. Experiments demonstrate that this approach reduces the grokking onset time by over 20-fold on modular addition while achieving 100% generalization; however, no acceleration is observed on the S5 task, highlighting the critical role of alignment between task symmetry and inductive bias. This study presents the first causal evidence—through targeted architectural intervention—that architectural freedom underlies grokking.
📝 Abstract
Mechanistic interpretability typically relies on post-hoc analysis of trained networks. We instead adopt an interventional approach: testing hypotheses a priori by modifying architectural topology to observe training dynamics. We study grokking - delayed generalization in Transformers trained on cyclic modular addition (Zp) - investigating if specific architectural degrees of freedom prolong the memorization phase. We identify two independent structural factors in standard Transformers: unbounded representational magnitude and data-dependent attention routing. First, we introduce a fully bounded spherical topology enforcing L2 normalization throughout the residual stream and an unembedding matrix with a fixed temperature scale. This removes magnitude-based degrees of freedom, reducing grokking onset time by over 20x without weight decay. Second, a Uniform Attention Ablation overrides data-dependent query-key routing with a uniform distribution, reducing the attention layer to a Continuous Bag-of-Words (CBOW) aggregator. Despite removing adaptive routing, these models achieve 100% generalization across all seeds and bypass the grokking delay entirely. To evaluate whether this acceleration is a task-specific geometric alignment rather than a generic optimization stabilizer, we use non-commutative S5 permutation composition as a negative control. Enforcing spherical constraints on S5 does not accelerate generalization. This suggests eliminating the memorization phase depends strongly on aligning architectural priors with the task's intrinsic symmetries. Together, these findings provide interventional evidence that architectural degrees of freedom substantially influence grokking, suggesting a predictive structural perspective on training dynamics.