An Optimal Agnostic PAC Algorithm

2026-08-06

AI Summary

This work resolves the problem of optimal sample complexity in agnostic PAC learning for hypothesis classes with finite VC dimension. By integrating VC theory, refined concentration inequalities, and the empirical risk minimization framework, it introduces the first algorithm that achieves the information-theoretic lower bound—up to a universal constant factor—for any optimal risk \(L^*\). The proposed method guarantees, with high probability, a risk upper bound of \(L(\hat{h}) \leq L^* + 7\cdot10^8\left(\sqrt{\frac{L^*(d+\log(1/\delta))}{n}} + \frac{d+\log(1/\delta)}{n}\right)\), which exactly matches the known minimax lower bound. This result establishes the first tight characterization of sample complexity across the full range of parameters in agnostic learning.

Abstract

Let $H\subseteq\{-1,+1\}^X$ be a class of finite VC dimension $d\ge1$. Writing $L$ for the binary risk and $L^*=\min_{h\in H}L(h)$, we construct a learner achieving the statistically optimal risk bound: from an i.i.d.\ sample of size $n$, for every $0<δ\le 1/2$, with probability at least $1-δ$, \[ L(\widehat h) \le L^*+ 7\cdot10^8\left( \sqrt{\frac{L^*(d+\log(1/δ))}{n}} +\frac{d+\log(1/δ)}{n} \right). \] This settles the sample complexity of agnostic PAC learning up to universal constants at every fixed $L^*$, matching the lower bounds of Devroye, Györfi, and Lugosi [A Probabilistic Theory of Pattern Recognition, Springer, 1996].

Problem

• agnostic PAC learning
• sample complexity
• VC dimension
• optimal risk bound
• statistical learning theory

Innovation

• agnostic PAC learning
• optimal risk bound
• VC dimension
• sample complexity
• statistical learning theory