Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods

2026-07-29

AI Summary

This study addresses the distortion in model evaluation caused by conventional random splitting, which often fails to preserve consistent data distributions—such as class imbalance, cluster structure, or spatial autocorrelation—between training and test sets. To mitigate this issue, the authors propose an Optimised-Distribution method that explicitly optimizes the similarity between the distributions of the training and test sets. The approach systematically integrates chi-squared tests, Kolmogorov-Smirnov tests, and Maximum Mean Discrepancy (MMD) as distributional similarity metrics. Evaluated across 15 UCI benchmark datasets against multiple splitting strategies—including random splitting, stratified sampling, Kennard-Stone, Duplex, and SPXY—the proposed method achieves an average MMD similarity of 89.0%, significantly outperforming existing techniques while enhancing both split quality and evaluation stability.

Abstract

Accurate model evaluation in machine learning depends critically on how datasets are split into training and testing subsets. Standard random splitting assumes that both partitions share the same underlying distribution, an assumption often violated in datasets with class imbalance, natural clustering, or spatial autocorrelation. This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation. Five established strategies are compared across fifteen UCI benchmark datasets: random splitting, stratified sampling, Kennard-Stone, Duplex, and SPXY. Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests. Geometry-based methods consistently produce near-zero MMD scores, introducing instability into downstream performance estimates. The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.

Problem

• train-test splitting
• statistical similarity
• AutoML
• model evaluation
• distribution mismatch

Innovation

• Optimised-Distribution
• train-test splitting
• Maximum Mean Discrepancy
• statistical similarity
• AutoML