distributional robustness

This study investigates why empirical risk minimization (ERM) can outperform specialized methods in certain out-of-distribution generalization settings, with a focus on the role of distributional shifts across training domains. Through theoretical analysis—including the derivation of error upper bounds—and systematic experiments, the work reveals for the first time that the strength of distributional shift itself is a critical factor in enhancing invariant prediction capabilities. Specifically, when the shift is sufficiently strong, ERM can closely approximate the ideal invariant predictor (Oracle) and even achieve comparable performance under certain conditions. These findings challenge the prevailing assumption that complex algorithms are necessary to attain invariance, and establish a quantitative relationship between the magnitude of distributional shift and out-of-distribution generalization performance.

Distribution shift in tabular data lacks empirical grounding, and existing robust learning methods—such as distributionally robust optimization (DRO)—frequently fail in real-world settings. Prior work predominantly assumes covariate shift (X-shift), whereas empirical analysis reveals that label-conditional shift (Y|X-shift) is both more prevalent and dominant in practice. Method: The authors construct a comprehensive benchmark platform spanning five real-world tabular datasets and 60,000 experimental configurations, enabling large-scale ablation studies on robustness. Contribution/Results: (1) Theory-driven methods like DRO offer no consistent advantage over empirical risk minimization (ERM); (2) implementation details—including model selection, hyperparameter tuning, and imbalance handling—exert far greater influence on robustness than the design of ambiguity sets; (3) a data-driven, inductive paradigm should supplant prior structural assumptions. These findings challenge the conventional paradigm in distributionally robust learning and provide reproducible, empirically grounded guidance for co-optimizing algorithms and data.

To address weak out-of-distribution (OOD) generalization, compromised in-distribution (ID) performance, and strong assumption dependence in existing causal approaches under distributional shift, this paper proposes Distributionally Robust Invariant Gradients (DRIG), a gradient-based distributionally robust method. DRIG models linear causal structures via generalized additive interventions and—uniquely—integrates data-dependent noise interventions with gradient invariance, establishing finite-sample robustness under class shifts through theoretical analysis. It unifies and extends the anchor regression framework to accommodate broader perturbation classes and further generalizes to semi-supervised domain adaptation. Experiments on synthetic data, single-cell transcriptomics, and intensive care datasets demonstrate that DRIG achieves superior trade-offs between OOD robustness and ID accuracy, consistently outperforming standard empirical risk minimization (ERM), distributionally robust optimization (DRO), and state-of-the-art causal baselines.

This work addresses the problem of evaluating the worst-case subgroup performance of machine learning models under distributional shift, to quantify their distributional robustness. For subgroups of fixed size defined by an arbitrary (continuous) core attribute (Z), we propose a two-stage estimation framework: first, conditionally estimate subgroup performance given (Z); second, optimize over (Z) to identify the worst-case subgroup. Unlike traditional Rademacher complexity-based approaches, our method avoids exponential dependence on the dimension of (Z), achieving dimension-free finite-sample convergence guarantees—where estimation error depends only on the conditional generalization error given (Z). Our key theoretical contribution is the first scalable, non-conservative statistical certification framework for robustness across intersecting vulnerable subgroups. Empirical evaluation on real-world datasets demonstrates the method’s effectiveness in identifying unreliable models and supporting robust deployment decisions.

Existing invariant feature learning methods for robustness under distribution shifts rely on the strong assumption that invariant features are fully observable in both training and test distributions—a condition frequently violated in practice, leading to poor generalization. Method: We propose Sufficient Invariant Learning (SIL), the first framework to formally define and model *sufficient invariant features*: the minimal subset of invariant features capable of independently supporting robust prediction. We theoretically prove that co-flat minima can accommodate diverse sufficient invariant features. Methodologically, we design Adaptive Sharpness-Aware Group Distributionally Robust Optimization (ASGDRO), which jointly enforces inter-environment flatness alignment and diversity regularization over invariant features. Results: SIL achieves significant improvements over state-of-the-art methods across multiple benchmarks and newly constructed datasets, demonstrating superior robustness against various types of distribution shifts.

When predictive models are deployed in new environments, their performance often degrades due to covariate shift and selective labeling, which jointly obscure accurate assessment of the true target risk. This work proposes an unbiased risk estimation method that integrates double machine learning with influence functions to simultaneously address both sources of bias for the first time. The approach is model-agnostic and compatible with general loss functions, constructing a corrected target risk estimator via nonparametric and plug-in techniques. Experiments on eICU electronic health record data demonstrate that the proposed method significantly outperforms baselines that handle only one type of bias or naively combine existing approaches, yielding more accurate tracking of the true target risk.

This work addresses the limitation of existing generalization bounds, which assume identical training and test distributions and thus fail to account for distributional shifts arising from varying proportions of calm and crisis regimes in environments with mechanism switches. Focusing on Markov-switching data distributions, the paper proposes a theoretical framework that quantifies the excess risk induced by mismatched regime compositions. The key contributions include extending generalization bounds to β-mixing sequences, introducing an effective sample size corrected by the spectral gap, and precisely decomposing the effects of regime mismatch and regime sensitivity. A matching minimax lower bound is also established. Empirical results on both synthetic data and 25 years of global equity indices demonstrate that the proposed penalty term significantly predicts ex post generalization gaps, whereas conventional estimators relying solely on the training set lack such predictive power.

This work addresses the sensitivity of Bayesian optimization to model misspecification, which can lead to fragile out-of-sample decisions. The authors propose a distributionally robust optimization framework that formalizes and quantifies model robustness under perturbations to both parameters and likelihood through novel measures: posterior sensitivity and likelihood sensitivity. Theoretical analysis reveals that posterior sensitivity vanishes as variance decreases, whereas likelihood sensitivity persists; parameter learning mitigates the former but cannot eliminate the latter. By constructing an uncertainty set based on a bias-aware divergence measure, the method achieves a near-Pareto-optimal trade-off between expected performance and dual robustness. Empirical experiments validate the effectiveness of the proposed approach.

Existing evaluation methods struggle to disentangle whether performance degradation under temporal distribution shifts stems from insufficient model adaptability or increased data difficulty. To address this, this work introduces a novel approach that decouples model adaptability from the inherent difficulty of temporal data for the first time. The authors propose three dynamic metrics based on performance trajectories, which capture the adaptation process through dynamic evaluation and comparative analysis. These new metrics uncover fine-grained adaptation patterns obscured by conventional assessment techniques, substantially enhancing the interpretability and depth of understanding of temporal robustness in machine learning models.

Data-driven decision-making often fails in deployment due to distributional shifts arising from contextual dependencies, partial observability, or stress perturbations. This work proposes a unified framework for flow- and score-based generative models that leverages pushforward mappings, velocity fields, and score fields, integrated with the Fokker–Planck equation and Wasserstein geometry to construct, transform, and optimize uncertainty distributions in probability space. By doing so, the approach elevates generative models from mere sample synthesis tools to principled mathematical mechanisms for distributional manipulation, enabling robustness analysis, conditional inference, and stress-scenario generation. The method is supported by theoretical guarantees, including forward–backward convergence, first-order minimax optimality in the space of transport maps, and error bounds for posterior sampling with generative priors.