Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory

2025-08-14

AI Summary

Traditional data splitting methods ignore intrinsic instance quality, undermining model validation robustness. This paper introduces Item Response Theory (IRT)—a psychometric framework—into the machine learning validation phase for the first time. We model instance heterogeneity using IRT’s three parameters: difficulty, discrimination, and guessing. Based on this, we propose an IRT-guided data partitioning method that explicitly accounts for instance-level reliability. Key findings reveal that high-guessing instances significantly degrade model performance and identify interpretable subgroups affecting the bias–variance trade-off. Experiments across four tabular datasets demonstrate that our optimized splitting improves validation accuracy by over 20 percentage points (e.g., rising from <50% to >70% in certain cases), substantially enhancing assessment reliability. This work establishes a novel, data-quality-aware paradigm for model validation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset.

Problem

• Improving ML validation by partitioning data based on instance quality
• Using Item Response Theory to identify informative data subgroups
• Reducing model bias and variance with IRT-informed balanced partitions

Innovation

• Uses Item Response Theory for data partitioning
• Creates balanced partitions to analyze bias-variance tradeoff
• Identifies high-guessing instances affecting model performance