Misalignment of Low-Loss Regions Causes Grokking

2026-09-30

AI Summary

This study addresses the unclear underlying mechanisms behind the delayed emergence of validation generalization in grokking, highlighting limitations in existing explanations. By constructing an analytical framework grounded in mode connectivity and the geometry of low-loss regions, this work reveals that the misalignment of low-loss regions induced by training-validation partitions is a critical cause of grokking. Furthermore, it proposes a symmetry-preserving data partitioning strategy to generate stable anti-grokking cases. These findings challenge prevailing correlation-based explanatory theories and demonstrate that when low-loss regions are aligned, hyperparameter tuning alone cannot induce grokking; instead, model dynamics collapse into either trainable or untrainable states.

Abstract

Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.

Problem

• Grokking
• generalization
• low-loss regions
• overfitting
• mode connectivity

Innovation

• Grokking
• Mode Connectivity
• Low-Loss Regions
• Misalignment
• Anti-Grokking