Efficient compression of neural networks and datasets

2025-05-23

AI Summary

This work addresses efficient, lossless compression of neural networks and datasets, aiming to drastically reduce model parameter counts while strictly bounding accuracy degradation. We propose a Monte Carlo–free probabilistic reconstruction framework based on ℓ₀ regularization, integrating a smooth ℓ₀ approximation, hierarchical pruning, and a teacher-student synthesis mechanism. For the first time, we establish a theoretical connection between neural compression and Solomonoff induction, with empirical validation. On CNNs (image classification) and Transformers (Wikipedia subset), our method achieves up to 90% parameter reduction with <1% test accuracy drop. Synthetic experiments demonstrate superior sample convergence of the proposed regularized model. Key contributions include: (i) a unified probabilistic modeling framework for neural compression; (ii) a novel differentiable paradigm for ℓ₀-norm optimization; and (iii) a foundational interpretation of compression as Solomonoff-style inductive inference.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We compare, improve, and contribute methods that substantially decrease the number of parameters of neural networks while maintaining high test accuracy. When applying our methods to minimize description length, we obtain very effective data compression algorithms. In particular, we develop a probabilistic reformulation of $ell_0$ regularized optimization for nonlinear models that does not require Monte-Carlo sampling and thus improves upon previous methods. We also improve upon methods involving smooth approximations to the $ell_0$ norm, and investigate layerwise methods. We compare the methods on different architectures and datasets, including convolutional networks trained on image datasets and transformers trained on parts of Wikipedia. We also created a synthetic teacher-student setup to investigate compression in a controlled continuous setting. Finally, we conceptually relate compression algorithms to Solomonoff's theory of inductive inference and empirically verify the prediction that regularized models can exhibit more sample-efficient convergence.

Problem

• Efficient neural network compression maintaining accuracy
• Improved probabilistic reformulation for nonlinear models
• Relating compression to inductive inference theory

Innovation

• Probabilistic reformulation of ell_0 regularization
• Improved smooth approximations to ell_0 norm
• Layerwise compression methods for neural networks