Information Consistent Pruning: How to Efficiently Search for Sparse Networks?

2025-01-26

AI Summary

Iterative Magnitude Pruning (IMP) suffers from low efficiency due to frequent retraining after each pruning step. Method: This paper proposes a theoretically grounded, dynamic stopping criterion based on inter-layer consistency between information flow and gradient flow, embedding the first information-gradient joint monitoring mechanism into the IMP framework to enable efficient sparse network search without intermediate retraining. Contribution/Results: The criterion accurately identifies pruning convergence, eliminating redundant iterations. Experiments across multiple dataset–DNN combinations demonstrate that our method accelerates IMP by 1.8–3.2× over state-of-the-art variants while preserving final sparse model accuracy. The implementation is publicly available.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Non-convex Optimization
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Iterative magnitude pruning methods (IMPs), proven to be successful in reducing the number of insignificant nodes in over-parameterized deep neural networks (DNNs), have been getting an enormous amount of attention with the rapid deployment of DNNs into cutting-edge technologies with computation and memory constraints. Despite IMPs popularity in pruning networks, a fundamental limitation of existing IMP algorithms is the significant training time required for each pruning iteration. Our paper introduces a novel extit{stopping criterion} for IMPs that monitors information and gradient flows between networks layers and minimizes the training time. Information Consistent Pruning (ourmethod{}) eliminates the need to retrain the network to its original performance during intermediate steps while maintaining overall performance at the end of the pruning process. Through our experiments, we demonstrate that our algorithm is more efficient than current IMPs across multiple dataset-DNN combinations. We also provide theoretical insights into the core idea of our algorithm alongside mathematical explanations of flow-based IMP. Our code is available at url{https://github.com/Sekeh-Lab/InfCoP}.

Problem

• Efficient Training
• Neural Network Pruning
• Simple Network Structure

Innovation

• Information Consistency Pruning
• Efficient Training Time
• Network Performance Retention