🤖 AI Summary
This study addresses the challenge of estimating training sample requirements and performance ceilings in machine learning by proposing Seer, a system that leverages learning curve modeling to predict both the sample size needed for target accuracy and the maximum achievable performance. Methodologically, Seer constructs an optimally constrained statistical model and employs an efficient maximum likelihood estimation algorithm, integrating nonlinear regression with iterative optimization for parameter inference. Extensive experiments across nearly one hundred real-world datasets spanning three domains demonstrate that the system accurately characterizes learning dynamics and forecasts model performance. Notably, its predictive accuracy approaches theoretical limits, thereby providing a reliable quantitative foundation for computational resource planning.
📝 Abstract
The research presented here focuses on modeling machine-learning performance. The thesis introduces Seer, a system that generates empirical observations of classification-learning performance and then uses those observations to create statistical models. The models can be used to predict the number of training examples needed to achieve a desired level and the maximum accuracy possible given an unlimited number of training examples. Seer advances the state of the art with 1) models that embody the best constraints for classification learning and most useful parameters, 2) algorithms that efficiently find maximum-likelihood models, and 3) a demonstration on real-world data from three domains of a practicable application of such modeling.
The first part of the thesis gives an overview of the requirements for a good maximum-likelihood model of classification-learning performance. Next, reasonable design choices for such models are explored. Selection among such models is a task of nonlinear programming, but by exploiting appropriate problem constraints, the task is reduced to a nonlinear regression task that can be solved with an efficient iterative algorithm. The latter part of the thesis describes almost 100 experiments in the domains of soybean disease, heart disease, and audiological problems. The tests show that Seer is excellent at characterizing learning-performance and that it seems to be as good as possible at predicting learning performance. Finally, recommendations for choosing a regression model for a particular situation are made and directions for further research are identified.