Growing Networks with Autonomous Pruning

2026-03-20

AI Summary

This work addresses the limitations of conventional convolutional neural networks, which often suffer from parameter redundancy and an inability to adaptively balance accuracy and efficiency during training. To overcome these challenges, the authors propose a dynamic network structuring method that simultaneously enhances model capacity through periodic network growth and maintains extreme parameter sparsity via gradient-driven autonomous pruning. This approach achieves, for the first time, synergistic optimization of network expansion and compression within a single training process, automatically yielding highly accurate yet sparse models without manual intervention. Experimental results demonstrate its efficacy: the method attains 99.44% accuracy on MNIST with only 6.2k parameters and achieves 92.2% accuracy on CIFAR-10 using just 157.8k parameters, significantly outperforming existing lightweight models.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2\ accuracy with 157.8k parameters.

Problem

• image classification
• parameter efficiency
• sparse neural networks
• model compression

Innovation

• autonomous pruning
• dynamic network growth
• parameter efficiency
• sparse neural networks
• gradient-based pruning