🤖 AI Summary
This paper addresses the non-monotonic phenomenon in machine learning where increased training data paradoxically degrades test performance. Grounded in PAC learning theory, it provides the first rigorous proof that, under the i.i.d. assumption, the test error of Empirical Risk Minimization (ERM) is monotonically non-increasing with sample size for any PAC-learnable problem with finite VC dimension or finite hypothesis space—establishing monotonicity as an intrinsic property of PAC learnability, not a special condition. The analysis integrates VC-dimension theory, sample complexity bounds, and statistical modeling, and is empirically validated to confirm consistency between theoretical lower bounds and observed risk decay. This result rectifies prior misconceptions treating monotonicity as contingent on additional constraints and furnishes foundational theoretical guarantees for designing robust learning algorithms. (149 words)
📝 Abstract
Monotone learning refers to learning processes in which expected performance consistently improves as more training data is introduced. Non-monotone behavior of machine learning has been the topic of a series of recent works, with various proposals that ensure monotonicity by applying transformations or wrappers on learning algorithms. In this work, from a different perspective, we tackle the topic of monotone learning within the framework of Probably Approximately Correct (PAC) learning theory. Following the mechanism that estimates sample complexity of a PAC-learnable problem, we derive a performance lower bound for that problem, and prove the monotonicity of that bound as the sample sizes increase. By calculating the lower bound distribution, we are able to prove that given a PAC-learnable problem with a hypothesis space that is either of finite size or of finite VC dimension, any learning algorithm based on Empirical Risk Minimization (ERM) is monotone if training samples are independent and identically distributed (i.i.d.). We further carry out an experiment on two concrete machine learning problems, one of which has a finite hypothesis set, and the other of finite VC dimension, and compared the experimental data for the empirical risk distributions with the estimated theoretical bound. The results of the comparison have confirmed the monotonicity of learning for the two PAC-learnable problems.