🤖 AI Summary
This work aims to derive tighter information-theoretic generalization error bounds to deepen the understanding of the generalization capability of randomized learning algorithms. By establishing a unified measure-change framework grounded in the data processing inequality for f-divergences, we propose a class of general and concise inequalities that flexibly adapt to diverse settings, including conditional mutual information, PAC-Bayes, and differential privacy. This approach not only simplifies and recovers several existing state-of-the-art results but also yields novel high-probability generalization bounds across multiple learning frameworks, significantly improving both the tightness and theoretical applicability of these bounds.
📝 Abstract
In this paper, we propose a novel class of change of measure inequalities via a unified framework based on the data processing inequality for $f$-divergences, which is surprisingly elementary yet powerful enough to yield tighter inequalities. We provide change of measure inequalities in terms of a broad family of information measures, including $f$-divergences (with Kullback-Leibler divergence and $\chi^2$-divergence as special cases), R\'enyi divergence, and $\alpha$-mutual information (with maximal leakage as a special case). We then embed these inequalities into the analysis of generalization error for stochastic learning algorithms, yielding novel and tighter high-probability information-theoretic generalization bounds, while also recovering several best-known results via simplified analyses. A key advantage of our framework is its flexibility: it readily adapts to a range of settings, including the conditional mutual information framework, PAC-Bayesian theory, and differential privacy mechanisms, for which we derive new generalization bounds.