How Does Machine Learning Manage Complexity?

2026-04-08

AI Summary

This study investigates the fundamental limits of machine learning models when handling data distributions that are uncomputable or of high computational complexity. Framing the learning process through the lens of computational complexity theory, the work abstracts it as generating P/poly-computable distributions with polynomially bounded maximum entropy, integrating tools from information theory, computability analysis, and pseudorandom generator theory. The central contribution lies in establishing, for the first time, a formal connection between the modeling capacity of machine learning systems and the complexity class P/poly under a maximum entropy constraint. Specifically, the paper proves that if a model’s output distribution minimally approximates the distribution produced by a cryptographic pseudorandom generator, then it must be close to the uniform distribution—thereby revealing an intrinsic relationship between learnability and computational complexity.

Abstract

We provide a computational complexity lens to understand the power of machine learning models, particularly their ability to model complex systems. Machine learning models are often trained on data drawn from sampleable or more complex distributions, a far wider range of distributions than just computable ones. By focusing on computable distributions, machine learning models can better manage complexity via probability. We abstract away from specific learning mechanisms, modeling machine learning as producing P/poly-computable distributions with polynomially-bounded max-entropy. We illustrate how learning computable distributions models complexity by showing that if a machine learning model produces a distribution $μ$ that minimizes error against the distribution generated by a cryptographic pseudorandom generator, then $μ$ must be close to uniform.

Problem

• machine learning
• computational complexity
• computable distributions
• complex systems
• probability

Innovation

• computational complexity
• computable distributions
• P/poly
• max-entropy
• pseudorandom generators