The Morgan-Pitman Test of Equality of Variances and its Application to Machine Learning Model Evaluation and Selection

2025-09-15

AI Summary

Existing nonlinear model selection predominantly relies on performance metrics while neglecting sampling variability and lacking rigorous statistical tests for homoscedasticity of prediction errors. To address this, we propose a robustified Morgan–Pitman test for variance equality, incorporating residual decorrelation preprocessing and heavy-tailed distribution–adaptive robust estimation. This approach markedly enhances robustness against high-variance outliers and non-normal errors without assuming any specific error distribution, making it broadly applicable to prediction error comparisons across arbitrary machine learning models. Simulation studies and real-data experiments demonstrate that the method maintains high statistical power and stringent type-I error control—even under small-sample, heteroscedastic, and heavy-tailed conditions. Consequently, it provides an interpretable, reproducible, and distribution-free statistical criterion for principled model selection.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Model selection in non-linear models often prioritizes performance metrics over statistical tests, limiting the ability to account for sampling variability. We propose the use of a statistical test to assess the equality of variances in forecasting errors. The test builds upon the classic Morgan-Pitman approach, incorporating enhancements to ensure robustness against data with heavy-tailed distributions or outliers with high variance, plus a strategy to make residuals from machine learning models statistically independent. Through a series of simulations and real-world data applications, we demonstrate the test's effectiveness and practical utility, offering a reliable tool for model evaluation and selection in diverse contexts.

Problem

• Assessing equality of variances in forecasting errors
• Enhancing robustness for heavy-tailed data and outliers
• Ensuring statistical independence of model residuals

Innovation

• Enhanced Morgan-Pitman test for variance equality
• Robust to heavy-tailed distributions and outliers
• Ensures statistical independence of model residuals