🤖 AI Summary
This paper addresses the challenge of multi-model-class evaluation by proposing the Model Class Selection (MCS) framework, which identifies the collection of model classes each containing at least one optimal model—thereby enabling formal comparison of performance equivalence across model classes of differing complexity (e.g., interpretable vs. black-box models). MCS generalizes conventional model selection and Model Set Selection (MSS) by integrating likelihood maximization and risk minimization criteria via a data-splitting strategy under mild assumptions. Theoretical analysis establishes its statistical validity. Empirical evaluation—including simulations and real-data experiments—demonstrates that MCS robustly identifies simple, interpretable model classes whose predictive performance matches that of complex models. By bridging interpretability and performance assessment, MCS introduces a novel paradigm and practical tool for explainable AI research.
📝 Abstract
Classical model selection seeks to find a single model within a particular class that optimizes some pre-specified criteria, such as maximizing a likelihood or minimizing a risk. More recently, there has been an increased interest in model set selection (MSS), where the aim is to identify a (confidence) set of near-optimal models. Here, we generalize the MSS framework further by introducing the idea of model class selection (MCS). In MCS, multiple model collections are evaluated, and all collections that contain at least one optimal model are sought for identification. Under mild conditions, data splitting based approaches are shown to provide general solutions for MCS. As a direct consequence, for particular datasets we are able to investigate formally whether classes of simpler and more interpretable statistical models are able to perform on par with more complex black-box machine learning models. A variety of simulated and real-data experiments are provided.