Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

2025-10-13

AI Summary

This study addresses the theoretical assessment of neural network compressibility limits. We extend the Minimum Description Length (MDL) principle—traditionally applicable only to regular models—to singular statistical models by integrating singular learning theory, and propose a novel model complexity estimator based on the Local Learning Coefficient (LLC). Systematic evaluation on the Pythia model family demonstrates a strong linear correlation between LLC and achievable compression ratios across diverse techniques, including quantization and tensor decomposition. Our approach yields the first computationally tractable, theoretically grounded complexity measure for neural networks, overcoming the fundamental limitation that conventional MDL is inapplicable to non-regular (singular) models. By establishing a principled, interpretable link between intrinsic model complexity and compressibility, this work provides a rigorous theoretical framework for characterizing fundamental compression limits in deep learning.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Learning & Optimization for NLP
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.

Problem

• Extending MDL principle to singular models like neural networks
• Investigating correlation between local learning coefficient and compressibility
• Rigorously evaluating theoretical limits of neural network compression

Innovation

• Extends MDL principle to singular neural network models
• Uses local learning coefficient to estimate model complexity
• Evaluates compression via quantization and factorization techniques