Lean Formalization of Generalization Error Bound by Rademacher Complexity

2025-03-25

AI Summary

This work addresses the formal verification of generalization error bounds in machine learning. Methodologically, it constructs, for the first time in Lean 4, an end-to-end verifiable theoretical framework based on Rademacher complexity: it formally defines empirical and population Rademacher complexities, proves the symmetrization lemma, and integrates McDiarmid’s inequality, Hoeffding’s lemma, and a verified probability inequality library to rigorously derive upper bounds on generalization error. In contrast to classical PAC and VC-dimension frameworks—which impose restrictive assumptions on hypothesis classes—this approach directly supports modern learning models, including deep neural networks and kernel methods. All definitions, lemmas, and theorems are type-checked in Lean 4, yielding the first fully formalized, mechanically verified, and reusable foundation for Rademacher-based generalization theory.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We formalize the generalization error bound using Rademacher complexity in the Lean 4 theorem prover. Generalization error quantifies the gap between a learning machine's performance on given training data versus unseen test data, and Rademacher complexity serves as an estimate of this error based on the complexity of learning machines, or hypothesis class. Unlike traditional methods such as PAC learning and VC dimension, Rademacher complexity is applicable across diverse machine learning scenarios including deep learning and kernel methods. We formalize key concepts and theorems, including the empirical and population Rademacher complexities, and establish generalization error bounds through formal proofs of McDiarmid's inequality, Hoeffding's lemma, and symmetrization arguments.

Problem

• Formalize generalization error bound using Rademacher complexity
• Quantify performance gap between training and test data
• Establish bounds via McDiarmid's inequality and symmetrization

Innovation

• Formalizing Rademacher complexity in Lean 4
• Proving generalization bounds via McDiarmid's inequality
• Applying Rademacher complexity to diverse ML scenarios