How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning

2024-07-08

arXiv.org

AI Summary

This paper addresses how deep neural networks (DNNs) overcome the “curse of dimensionality” in high-dimensional learning. Method: It proposes a modeling paradigm grounded in function composition structures (h∘g) and symmetry quotient mappings, integrating covering number theory with F₁-norm-based learnability characterization, and combining Sobolev/Barron norm-constrained optimization with symmetry-driven dimensionality reduction. Contribution/Results: The work establishes novel generalization bounds for composite functions under symmetry, rigorously proving that DNN global minima achieve ε-accuracy approximation using only O(1/ε²) samples. It uncovers a phase transition mechanism arising from asymmetric learning—where g performs smooth dimensionality reduction and h exhibits low regularity—and experimentally validates both the predicted phase transition and dimension-independent convergence rates. These results provide new theoretical foundations for sample-efficient, high-dimensional learning with DNNs.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

We show that deep neural networks (DNNs) can efficiently learn any composition of functions with bounded $F_{1}$-norm, which allows DNNs to break the curse of dimensionality in ways that shallow networks cannot. More specifically, we derive a generalization bound that combines a covering number argument for compositionality, and the $F_{1}$-norm (or the related Barron norm) for large width adaptivity. We show that the global minimizer of the regularized loss of DNNs can fit for example the composition of two functions $f^{*}=hcirc g$ from a small number of observations, assuming $g$ is smooth/regular and reduces the dimensionality (e.g. $g$ could be the quotient map of the symmetries of $f^{*}$), so that $h$ can be learned in spite of its low regularity. The measures of regularity we consider is the Sobolev norm with different levels of differentiability, which is well adapted to the $F_{1}$ norm. We compute scaling laws empirically and observe phase transitions depending on whether $g$ or $h$ is harder to learn, as predicted by our theory.

Problem

• DNNs break dimensionality curse via function composition learning.
• Generalization bound combines covering number and F1-norm adaptivity.
• DNNs fit composite functions from limited observations efficiently.

Innovation

• DNNs break dimensionality curse via compositionality.
• Generalization bound combines covering number and F1-norm.
• Empirical scaling laws show phase transitions in learning.