The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting

2026-07-31

AI Summary

This work uncovers a hidden vulnerability to catastrophic forgetting in neural networks that exhibit seemingly perfect generalization. Leveraging the grokking phenomenon in modular arithmetic, the study contrasts knowledge retention under random label noise between models trained with the AdamW optimizer and those sampled from high-entropy solution spaces. It introduces the concept of the “grokked illusion,” arguing that models residing in high-entropy regions demonstrate superior robustness. Methodologically, the authors combine Wang–Landau molecular dynamics to sample high-entropy models, standard AdamW training as a baseline, and singular value decomposition to analyze the effective rank of weight matrices. Experiments reveal that high-entropy models retain approximately 95% accuracy on original tasks after learning new ones—significantly outperforming AdamW-trained models (<75%)—thereby demonstrating that high-entropy solutions effectively mitigate catastrophic forgetting. The effective rank is further linked to the richness of feature representations, offering mechanistic insight into this resilience.

Abstract

While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is. Recent studies have shown that solutions occupying larger volumes in parameter space, as quantified by Boltzmann entropy, often exhibit superior generalizability compared to those reached by conventional optimization, a phenomenon known as the high entropy advantage. Here we ask whether this advantage persists beyond generalization. Specifically, we investigate models' robustness, the ability to retain the learned knowledge when the model is subsequently trained to acquire new information. Using grokking in modular arithmetic as a controlled setting, we design a noise injection experiment to evaluate the robustness difference between AdamW-trained transformers and high-entropy model sampled from Wang-Landau Molecular Dynamics with identical saturated performance. By forcing both models to fully remember new data with random labels, we find that AdamW-trained models suffer from catastrophic forgetting, with original task test accuracy dropping from 100% to below 75%, whereas the high-entropy models maintain approximately 95% test accuracy. We term this hidden fragility behind apparent generalization the "grokked illusion." Through singular value decomposition of the neural network weights, we discover that high-entropy neural networks possess significantly higher effective rank in attention and MLP layers both before and after noise injection, indicating richer feature representations can serve as a buffer against catastrophic forgetting. Our findings demonstrate that perfect generalization does not imply equal robustness, offering a new perspective on what makes a trained model robust to interference.

Problem

• catastrophic forgetting
• robustness
• generalization
• grokking
• neural network

Innovation

• high entropy advantage
• catastrophic forgetting
• grokking
• effective rank
• Wang-Landau Molecular Dynamics