Who Wins Where? Conformal Model Comparison for Local Superiority

2026-07-31

AI Summary

Traditional model comparison typically identifies only a globally superior model, overlooking heterogeneity in performance across the covariate space. This work proposes a conformal local model comparison framework that employs triple data splitting to estimate the center and scale of local scores and constructs one-sided conformal bounds with finite-sample marginal coverage guarantees, enabling calibrated identification of regions where one model significantly outperforms another. Theoretically, the method establishes consistency of local mean estimation and demonstrates that the globally optimal model may substantially diverge from locally dominant ones. Methodologically, under squared loss, it introduces a bias–variance decomposition to elucidate how model structure influences local superiority. Experiments on both synthetic and real data show that the approach accurately recovers heterogeneous superiority regions, abstains judiciously in high-uncertainty areas, and achieves higher conditional utility than global model selection.

Abstract

Standard model comparison is global, aggregating losses across the covariate space to declare a single winner. This can obscure heterogeneous performance, where different models are preferable in different regions. We introduce conformalized local model comparison, a split-sample framework for constructing calibrated local best-model maps. Given a model comparison score, such as the difference between two squared losses, the method uses three disjoint splits to fit competing models, estimate local centers and scales from out-of-sample scores, and conformally calibrate residual uncertainty. At a target point, the procedure declares a local winner only when a one-sided conformal bound excludes a tie, with the score's sign determining the favored model. We prove finite-sample marginal control for one-sided erroneous declarations on the realized future comparison score, establish pointwise consistency of the localized mean-score estimator away from tie boundaries, show that aggregate comparison can disagree sharply with the prevalence of local superiority, and derive a squared-loss bias--variance decomposition that clarifies how model structure affects local wins. Synthetic and real-data experiments show that the method recovers heterogeneous winner regions, abstains under uncertainty, and yields higher conditional gain than global selection.

Problem

• model comparison
• local superiority
• heterogeneous performance
• conformal inference
• covariate space

Innovation

• conformal inference
• local model comparison
• heterogeneous performance
• split-sample calibration
• conditional model selection