Simplifying Neural Networks During Training

2026-07-30

AI Summary

This work addresses the challenge of efficiently simplifying over-parameterized deep neural networks during training by proposing an online network simplification strategy grounded in training dynamics. The method uniquely integrates the geometric characteristics of neural collapse with layer importance analysis, employing the inverse Fisher criterion as an efficient proxy metric to quantify feature collapse. This enables automatic identification of both the boundary between feature extraction and classification and the optimal timing for simplification. Upon detection, the latter portion of the network is replaced with a lightweight classification head, and training proceeds uninterrupted. Evaluated on image classification tasks across MLP, VGG, and ResNet architectures, the approach substantially reduces model parameters while preserving accuracy comparable to that of the original full-sized models.

Abstract

Understanding and exploiting the training dynamics of overparameterized deep neural networks remains a central challenge in modern machine learning. Recent evidence on Neural Collapse (NC) shows that class representations and classifiers exhibit highly structured geometry, while the Tunnel Effect suggests that only a subset of layers is essential for feature extraction. We combine these two perspectives and propose an NC-inspired training framework for simplifying deep networks during training. Our method monitors representation dynamics through the Inverse Fisher Criterion, a stable and efficient proxy for the variability collapse behavior, to identify both the split point between feature extraction and classification and the training stage at which simplification becomes viable. We then replace the trailing layers with a lightweight classification head and continue training the reduced model. Experiments on image-classification benchmarks across MLP, VGG, and ResNet architectures show that the proposed method achieves substantial parameter reductions while maintaining accuracy comparable to that of the full model. Code to reproduce the experiments can be found at: https://github.com/LorenzoSciandra/NNS.

Problem

• Neural Collapse
• model simplification
• overparameterized networks
• training dynamics
• representation geometry

Innovation

• Neural Collapse
• Tunnel Effect
• Inverse Fisher Criterion
• Network Simplification
• Representation Dynamics