model selection

Addressing the challenge of robust prediction in high-dimensional data corrupted by outliers, this paper proposes a sparse, diverse, and robust ensemble modeling framework. Methodologically, we establish, for the first time, the finite-sample breakdown point theory for ensemble models; design a custom L₀-optimization-driven algorithm that jointly controls sparsity, diversity, and robustness; and integrate robust statistical modeling, ensemble learning, and cross-validation-based hyperparameter tuning. Our key contribution lies in introducing L₀ regularization into robust ensemble learning—enabling synergistic enhancement of outlier identification and outlier resistance. Experiments on contaminated bioinformatics and cheminformatics datasets demonstrate that the proposed method significantly outperforms existing single-model sparse robust approaches, achieving higher predictive accuracy and more reliable outlier detection.

This work addresses the susceptibility of feature selection on small-to-medium scientific datasets to instability and optimistic bias induced by data leakage, which mutually exacerbate one another. The authors propose the first unified framework that integrates bootstrap-based stability selection with rigorous nested cross-validation, performing preprocessing and feature selection entirely within each inner fold to simultaneously yield highly stable feature subsets and unbiased performance estimates. Centered on stability as the primary optimization objective, the method supports binary classification, multiclass classification, and regression tasks, incorporates nine diverse algorithms, and includes a deterministic test suite to ensure reproducibility. Evaluated on three real-world datasets, the approach achieves predictive performance comparable to the best baselines while demonstrating significantly higher Jaccard stability than ANOVA F-test, recursive feature elimination, and Boruta.

Traditional ensemble methods (e.g., Bagging, Boosting) suffer from high computational overhead and poor adaptability to heterogeneous data distributions. To address these limitations, we propose Hellsemble—a computationally efficient, progressive ensemble framework for binary classification. Hellsemble first partitions instances into dynamic “difficulty tiers” based on instance-level difficulty estimation; a learned router then progressively routes challenging samples across tiers to specialized base learners, enabling model specialization and load balancing. Its core innovations are a difficulty-driven dynamic routing mechanism and a progressive training paradigm, jointly enhancing interpretability, generalization, and inference efficiency. Evaluated on the OpenML-CC18 and Tabzilla benchmarks, Hellsemble consistently outperforms state-of-the-art ensemble methods, achieving an average accuracy gain of 2.1% and accelerating inference by 3.4×.

This study addresses the lack of statistical rigor in evaluating medical AI models across patient- and imaging-based stratifications. We propose the first end-to-end subgroup analysis framework. Methodologically, it integrates Bayesian posterior estimation with bootstrap resampling to quantify metric uncertainty, applies the Benjamini–Hochberg procedure for multiplicity correction, and introduces an adaptive subgroup scanning algorithm to automatically detect statistically significant and clinically relevant performance disparities. The framework explicitly handles challenges including small-sample subgroups, base-rate imbalance, and intersectional subgroup testing. Evaluated on ISIC2020 (skin lesion classification) and MIMIC-CXR (chest radiograph diagnosis), it successfully identifies interpretable fairness-critical subgroups—e.g., demographic or anatomical cohorts exhibiting systematic underperformance—thereby substantially enhancing both the statistical reliability and clinical interpretability of model evaluation.

This paper addresses the challenge of downstream adaptation for large language models when original training data is inaccessible. We propose a **non-parametric knowledge transfer paradigm**: rather than updating model weights, our approach extracts structured cognitive strategies from teacher model outputs via inference trajectory distillation and implicit behavioral modeling. The method comprises four core components: trajectory contrastive learning, latent state-space alignment, logical formalization distillation, and backpropagation-free policy imitation—constituting the first zero-gradient, memory-efficient knowledge absorption framework. Evaluated on six cross-task generalization benchmarks, our method achieves an average accuracy improvement of 9.2% and reduces inference latency by 37%, significantly outperforming parameter-efficient fine-tuning baselines (e.g., LoRA, QLoRA) and prompt engineering approaches.

This work addresses the challenge of achieving adaptive model selection that simultaneously ensures finite-sample coverage validity and accommodates spatial heterogeneity without distributional assumptions. The authors propose a localized conformal model selection framework that, during the calibration phase, symmetrically constructs upper and lower surrogate intervals to form a data-dependent safety index set for identifying favorable models. This approach enables local adaptivity while preserving sample exchangeability. Notably, it is the first method to unify local adaptivity with post-selection validity, guaranteeing exact marginal coverage while flexibly adapting to data heterogeneity. Empirical results demonstrate that under strong heterogeneity or low-noise conditions, the resulting prediction intervals are significantly shorter than those produced by a fixed oracle model.

This work proposes a novel ensemble framework termed Behavioral Profiling Ensemble, which addresses the limitation of traditional ensemble methods that assign static weights based on model diversity and validation sets, thereby ignoring the varying capabilities of base models across different regions of the input space. The key innovation lies in introducing the concept of “intrinsic behavioral profiles” for each base model, capturing its characteristic response patterns. During inference, ensemble weights are dynamically allocated according to the deviation between the test instance and each model’s behavioral profile. Notably, the approach eliminates the need for validation sets or assumptions about model diversity. Extensive experiments on multiple synthetic and real-world datasets demonstrate that the proposed method consistently outperforms existing ensemble techniques, achieving substantial improvements in predictive accuracy, computational efficiency, and memory overhead.

This work addresses the challenge that existing model evaluation methods often fail to reliably assess estimator quality in low-variance settings due to confounding between bias and variance or excessive sensitivity of statistical tests. To overcome this limitation, the authors propose a fault-tolerant evaluation framework that unifies bias and variance modeling through an adjustable tolerance parameter ε, enabling robust assessment of sample-efficient performance estimators within practically acceptable error margins. The framework integrates bias-variance analysis, fault-tolerant evaluation theory, and an adaptive ε-optimization algorithm, making it particularly well-suited for scenarios with low annotation costs. Experimental results demonstrate that the proposed approach provides a more comprehensive and reliable characterization of estimator behavior, significantly enhancing both the practical utility and stability of performance evaluation.

This study investigates whether model robustness must be explicitly optimized or whether it naturally emerges as task performance improves. Through systematic evaluation across multiple models and datasets—incorporating variables such as input perturbations, prompt rewrites, and temperature scaling—the authors find a strong positive correlation between task performance and robustness: as models approach performance saturation on a given task, their robustness also stabilizes. These findings suggest that robustness is not an independent property requiring separate optimization, but rather a natural byproduct of enhanced task-specific capabilities. This challenges the prevailing paradigm in the field that treats robustness as a distinct objective to be pursued independently of overall task performance.

This work proposes a data-aware sensitivity analysis framework for decision tree ensembles, addressing the lack of interpretable evidence grounded in realistic data distributions in existing approaches. By constraining perturbed samples to remain close to the training data distribution, the method generates more realistic and interpretable sensitivity examples to assess model robustness and fairness. It achieves, for the first time, data-aware verification of large-scale, multi-class tree ensembles—such as those with depth 8 and 800 trees—while strengthening the theoretical NP-hard foundation of sensitivity verification. Integrating mixed-integer linear programming (MILP), SMT encoding, and data-aware search optimization, the approach significantly improves verification efficiency and scalability, enabling reliable analysis in high-stakes applications.