Starting Off on the Wrong Foot: Pitfalls in Data Preparation

2026-03-18

AI Summary

This study addresses the limitations of conventional random splitting in highly imbalanced insurance loss data, which often compromises model statistical validity and stability. To overcome this, we propose a novel data preparation framework that integrates distribution-consistent train-test partitioning via support point sampling, nonparametric feature selection using the Chatterjee correlation coefficient, and unified missing value handling—all embedded within our custom InsurAutoML pipeline. This work represents the first systematic incorporation of support points and the Chatterjee correlation coefficient into the preprocessing phase of insurance modeling. Empirical evaluations on both synthetic and real-world datasets demonstrate that the proposed approach significantly enhances model robustness and interpretability while simultaneously reducing computational overhead.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Natural Language Processing: Safety and Robustness
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

When working with real-world insurance data, practitioners often encounter challenges during the data preparation stage that can undermine the statistical validity and reliability of downstream modeling. This study illustrates that conventional data preparation procedures such as random train-test partitioning, often yield unreliable and unstable results when confronted with highly imbalanced insurance loss data. To mitigate these limitations, we propose a novel data preparation framework leveraging two recent statistical advancements: support points for representative data splitting to ensure distributional consistency across partitions, and the Chatterjee correlation coefficient for initial, non-parametric feature screening to capture feature relevance and dependence structure. We further integrate these theoretical advances into a unified, efficient framework that also incorporates missing-data handling, and embed this framework within our custom InsurAutoML pipeline. The performance of the proposed approach is evaluated using both simulated datasets and datasets often cited in the academic literature. Our findings definitively demonstrate that incorporating statistically rigorous data preparation methods not only significantly enhances model robustness and interpretability but also substantially reduces computational resource requirements across diverse insurance loss modeling tasks. This work provides a crucial methodological upgrade for achieving reliable results in high stakes insurance applications.

Problem

• data preparation
• imbalanced insurance data
• train-test splitting
• statistical validity
• insurance loss modeling

Innovation

• support points
• Chatterjee correlation coefficient
• data preparation
• imbalanced insurance data
• InsurAutoML