🤖 AI Summary
Existing formulations of the fundamental theorem of agnostic PAC learning implicitly assume measurability conditions without explicitly specifying the minimal requirements for its rigorous validity.
Method: Grounded in measure theory, this work systematically identifies and makes explicit the minimal measurability assumptions necessary for the theorem’s strict validity. It integrates VC theory, model theory—particularly NIP and o-minimal structures—and statistical learning theory to deliver, for the first time, a self-contained, measure-theoretically rigorous proof.
Contributions: (1) A precise statement and fully rigorous proof framework for the foundational theorem; (2) Sufficient conditions for PAC learnability of hypothesis classes over o-minimal structures; (3) A proof that binary-classification neural networks with standard activation functions—including ReLU and sigmoid—are PAC learnable under o-minimal expansions, thereby establishing a solid measure-theoretic foundation for deep learning theory.
📝 Abstract
The Fundamental Theorem of Statistical Learning states that a hypothesis space is PAC learnable if and only if its VC dimension is finite. For the agnostic model of PAC learning, the literature so far presents proofs of this theorem that often tacitly impose several measurability assumptions on the involved sets and functions. We scrutinize these proofs from a measure-theoretic perspective in order to explicitly extract the assumptions needed for a rigorous argument. This leads to a sound statement as well as a detailed and self-contained proof of the Fundamental Theorem of Statistical Learning in the agnostic setting, showcasing the minimal measurability requirements needed. As the Fundamental Theorem of Statistical Learning underpins a wide range of further theoretical developments, our results are of foundational importance: A careful analysis of measurability aspects is essential, especially when the theorem is used in settings where measure-theoretic subtleties play a role. We particularly discuss applications in Model Theory, considering NIP and o-minimal structures. Our main theorem presents sufficient conditions for the PAC learnability of hypothesis spaces defined over o-minimal expansions of the reals. This class of hypothesis spaces covers all artificial neural networks for binary classification that use commonly employed activation functions like ReLU and the sigmoid function.