The optimal information complexity of VC learning

2026-10-06

AI Summary

This study addresses the limitation of existing conditional mutual information (CMI) frameworks in recovering optimal PAC generalization bounds for VC classes. We propose a novel algorithm based on a majority vote mechanism over randomized base learners. In the realizable setting, this algorithm strictly constrains the CMI to O(d), providing the first proof that optimal PAC guarantees for VC classes can be recovered via algorithm-dependent information complexity. Theoretical analysis demonstrates that our method achieves optimal expected generalization error bounds, thereby validating the feasibility of deriving tight generalization bounds within the CMI framework.

Abstract

Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning algorithms based on algorithm-dependent information-theoretic quantities. We study one of these quantities, the evaluated Conditional Mutual Information (eCMI). It has been an interesting question whether the optimal PAC guarantee for VC classes can be recovered from the algorithm-dependent analyses via CMI. And we show that it is possible to recover this guarantee by constructing a learning algorithm whose eCMI is of order O(d) in the realizable case, where d is the VC-dimension of the concept class. Specially, our algorithm is a randomized Majority-of-5 base learners with optimal in-expectation generalization guarantee.

Problem

• Conditional Mutual Information
• information complexity
• VC learning
• PAC guarantee
• eCMI

Innovation

• Conditional Mutual Information
• VC learning
• PAC guarantee
• information complexity
• randomized majority vote