Generalization in Neural Networks Through the Lens of Magnitude Potential

2026-10-01

AI Summary

This study addresses the limited interpretability of generalization mechanisms and training dynamics in neural networks. We introduce the concept of "magnitude potential" and propose a novel magnitude-based metric, computed as the ratio of intra-class to global magnitude potentials at the logit layer, to quantify representation quality and geometric structural properties of data points. Notably, this metric remains effective even when neural collapse is suppressed, revealing the geometric essence of generalization. Experimental results demonstrate that the proposed ratio correlates significantly with Feldman's memorization scores and accurately detects decision boundary shifts as well as grokking phenomena in modular arithmetic tasks. This work provides new theoretical perspectives and analytical tools for understanding deep learning generalization.

Abstract

Explaining generalization and training dynamics in neural networks remains a challenge, and various approaches have been developed to study different aspects of these phenomena. In this paper, we introduce the idea of {\em magnitude potential} -- a quantity based on the theory of metric magnitude -- that reflects how well an arbitrary point is represented by a given set. We find that this basic quantity can be applied to examine various features in neural generalization. The ratio between the magnitude potential with respect to a class and with respect to the entire data, computed at the logit layer, is informative of the representation of the point. In experiments, these ratios for individual training points are found to be correlated with the Feldman memorization scores. Magnitude potential ratios aggregated across points detect structural changes in the decision boundaries and provide a geometric indicator of grokking in modular arithmetic. Although the magnitude potential ratio and neural collapse are both closely associated with intra-class and inter-class geometric structure, the magnitude potential ratio remains informative even when neural collapse is explicitly suppressed.

Problem

• generalization
• training dynamics
• memorization
• grokking
• neural collapse

Innovation

• Magnitude Potential
• Generalization
• Grokking
• Metric Magnitude
• Neural Collapse