gradient-boosted trees

This work addresses the rigidity in model selection and poor interpretability inherent in conventional XGBoost–neural network ensembles. We propose an adaptive fusion framework grounded in dynamic meta-learning, which jointly leverages uncertainty quantification and feature importance as dual control signals to guide fine-grained scheduling and weighted integration of the two base models at inference time. Our key innovation lies in co-modeling uncertainty estimates and interpretability-aware metrics—specifically, feature importance—within the meta-learner’s decision process, thereby simultaneously enhancing predictive performance and decision transparency. Extensive experiments across multiple benchmark datasets demonstrate that our method consistently outperforms static ensembles and individual baselines, achieving average accuracy gains of 2.1–4.7 percentage points. Moreover, it provides auditable, instance-level rationale for model selection and feature-level attribution, supporting both reliability assessment and human-understandable explanations.

This work addresses the challenge of efficiently generating mixed-type tabular data by proposing XGenBoost, the first framework to effectively adapt XGBoost for generative modeling. The approach comprises two models: an XGBoost-driven denoising diffusion implicit model (DDIM) tailored for small datasets and a hierarchical autoregressive model designed for large-scale data. A key innovation is a Gaussian–multinomial joint diffusion mechanism that operates without one-hot encoding, combined with empirical quantile function-based dequantization and hierarchical classifiers to preserve the ordinal structure of numerical features. Evaluated across multiple benchmarks, XGenBoost consistently outperforms existing neural and tree-based generative models in generation quality while substantially reducing training costs.

This paper identifies a systemic issue in machine learning: preprocessing hyperparameters—such as missing-value imputation strategies—are frequently overlooked yet substantially bias model evaluation. Current practice often involves informal, post-hoc tuning of preprocessing steps, leading to optimistic performance estimates and irreproducible results. To address this, the authors formally distinguish and empirically analyze the coupling effects between algorithmic and preprocessing hyperparameters. Using a modular supervised learning workflow model, controlled variable experiments, replication of canonical case studies, and bias diagnostics, they quantify the resulting optimistic bias. Key contributions include: (1) establishing preprocessing hyperparameters as equally critical as algorithmic ones; (2) proposing formal modeling principles to eliminate informal preprocessing tuning; and (3) delivering actionable reporting guidelines for ML practitioners, thereby significantly enhancing model credibility and reproducibility.

This study addresses the critical dependence of tree boosting models’ generalization performance on hyperparameter configuration, a domain lacking systematic comparison and guiding principles among existing tuning methods. Conducting a large-scale empirical evaluation across 59 regression and classification datasets under a unified experimental framework, the authors compare prominent hyperparameter optimization approaches—including random search, grid search, Tree-structured Parzen Estimator (TPE), Gaussian process Bayesian optimization, Hyperband, and SMAC. The findings reveal that SMAC consistently outperforms other methods across most tasks, while default hyperparameters yield suboptimal results. Notably, all hyperparameters significantly influence performance, contradicting the common assumption that only a few are critical. Effective tuning requires extensive sampling (>100 trials), and for regression tasks, employing early stopping proves superior to including the number of iterations within the search space.

In same-domain repeated access scenarios, decision trees and their ensembles (e.g., random forests, GBDTs) struggle to jointly optimize accuracy and interpretability. Method: This paper proposes a learnable, tunable unified tree framework. It introduces (1) a parameterized splitting criterion that continuously interpolates between entropy and Gini impurity, enabling data-adaptive optimal splits; (2) a theoretical characterization of sample complexity, providing generalization guarantees for the interpretability–accuracy trade-off; and (3) joint optimization of Bayesian decision trees, minimum-cost-complexity pruning, and ensemble hyperparameters. Results: Extensive experiments on real-world datasets demonstrate that the framework significantly improves the consistency between predictive accuracy and model interpretability. It offers both theoretical rigor—via provable generalization bounds—and practical utility—through end-to-end differentiability and seamless integration into existing tree-based pipelines. The approach bridges a critical gap between statistical performance and human-understandable structure in tree learning.

This work addresses a critical oversight in modern tabular learning benchmarks—the neglect of feature engineering—which introduces bias in model evaluation and obscures its pivotal role in real-world applications. To bridge this gap, the authors propose TabPrep, a lightweight, domain-knowledge-driven preprocessing pipeline that systematically enhances model performance by identifying and transforming three fundamental structural data patterns through targeted feature generators. TabPrep integrates systematic feature engineering into mainstream tabular benchmarking for the first time, demonstrating that well-designed preprocessing alone can surpass performance gains from most architectural innovations. Evaluated on the TabArena benchmark, TabPrep consistently boosts the accuracy of diverse models—including tree-based methods, neural networks, linear models, and foundation models—while maintaining low computational overhead and strong generalizability.

This study investigates optimal parallel heterogeneous ensemble strategies for small- to medium-scale tabular classification tasks. Through systematic experiments on 56 OpenML CC18 datasets, the authors find that the instability of Blending and Stacking is mutually independent, and that Robust Soft Voting significantly outperforms Hard Voting in multi-class settings. Building on these insights, they propose a robust ensemble strategy and validate its effectiveness on the TabArena platform. The recommended approach significantly surpasses the Single Best method across 28 additional tasks and matches or exceeds the performance of all individual ensemble techniques considered.

This work proposes PINE, a novel tree ensemble pruning method that introduces conformal prediction to preserve prediction equivalence within a distributionally defined in-distribution region parameterized by α. Unlike conventional pruning approaches that trade prediction consistency for compression gains, or existing fidelity-preserving methods that maintain global prediction equivalence at the cost of limited compression efficiency, PINE leverages conformal calibration to construct in-distribution regions, designs a fidelity-aware pruning algorithm, and incorporates an equivalence verification mechanism. This enables substantial improvements in compression rates while strictly preserving predictions in critical regions. Experiments on twelve public tabular datasets demonstrate that PINE achieves up to 30% higher compression ratios compared to current fidelity-preserving pruning techniques, without compromising predictive fidelity.

This work addresses the high computational cost and deployment inefficiency of deep ensemble models on large-scale tabular data. The authors propose TabLoRA, the first approach to integrate low-rank adaptation (LoRA) into neural ensemble learning for tabular data. By sharing a common backbone network and equipping each predictor with a lightweight, trainable low-rank adapter module, TabLoRA preserves ensemble diversity without replicating backbone parameters. This design substantially reduces both parameter count and computational overhead. Empirical results demonstrate that TabLoRA outperforms state-of-the-art gradient-boosted decision tree (GBDT) methods and existing deep learning approaches across multiple benchmark datasets, achieving superior predictive performance while maintaining high efficiency under identical resource constraints.

This study addresses the limitations of current tabular foundation models (TFMs), which, despite strong individual performance, exhibit constrained generalization, while conventional ensemble methods suffer from model redundancy and calibration degradation. The authors systematically evaluate six modern TFMs across 153 OpenML classification tasks using six ensemble strategies, quantifying diversity via the Q-statistic and conducting comprehensive comparisons through Friedman–Nemenyi tests and calibration analysis. Their findings reveal an upper bound on ensemble diversity among TFMs (mean Q = 0.961), with most ensembles yielding only marginal accuracy gains (+0.18%) at a substantial computational cost (253× increase) and often impairing probability calibration. Among the strategies examined, greedy selection achieves the best trade-off between performance and efficiency, emerging as a practical default for TFM ensembling.