excess risk analysis

Existing H-consistency bounds rely heavily on strong convexity assumptions, limiting their applicability and yielding loose guarantees. Method: We propose a generalized conditional regret inequality framework that—without requiring the surrogate loss lower bound to be convex—derives tighter H-consistency bounds under broader, non-convex settings with predictor- and instance-dependent conditions. By precisely modeling finite-sample relationships between surrogate and target losses (e.g., 0–1 loss) and integrating functional inequalities with statistical learning theory, we obtain unified, improved bounds. Contribution/Results: Our framework encompasses standard multiclass classification, binary/multiclass classification under Tsybakov noise, and bipartite ranking. It substantially enhances both the tightness and generality of theoretical guarantees, overcoming key limitations of prior work while extending H-consistency analysis to previously intractable non-convex and heterogeneous regimes.

This work addresses the misalignment between offline evaluation metrics and online performance objectives in industrial applications by establishing a unified theoretical framework that systematically quantifies the relationships among diverse evaluation metrics for the first time. By introducing the concepts of Bayes-optimal sets and regret transfer mechanisms, the study reveals structural asymmetries among metrics and provides a principled classification and relational modeling of metrics with varying mathematical forms. Theoretically characterizing metric consistency and transferability, this research offers novel insights and a methodological foundation for designing offline evaluation systems that are aligned with online objectives and backed by rigorous theoretical guarantees.

This work establishes high-probability regret bounds for empirical risk minimization (ERM) and extends them to learning problems involving nuisance components, such as causal inference, missing data, and domain adaptation. By employing a three-step approach—elementary inequality, localized uniform concentration bounds, and a fixed-point argument—combined with a key radius defined via local Rademacher complexity, the study characterizes convergence rates in a modular analytical framework. This framework unifies the treatment of standard and nuisance-augmented ERM, explicitly decomposing statistical and approximation errors, and provides sufficient conditions for fast convergence. It recovers classical rates for VC-subgraph classes, Sobolev/Hölder spaces, and bounded variation function classes, and delivers transferable regret guarantees for orthogonal learning settings.

In combinatorial optimization, the empirical risk w.r.t. model parameters is piecewise constant, hindering gradient-based optimization and lacking theoretical generalization guarantees. Method: For contextual stochastic optimization with complex objectives, we propose a perturbation-driven risk smoothing strategy. Our approach integrates statistical learning models with a surrogate combinatorial optimization oracle to construct a context-aware, generalization-controllable decision framework. Contribution/Results: We establish the first unified generalization bound incorporating perturbation bias, statistical error, and optimization error. We introduce the notion of “uniform weak consistency” to characterize the coupled stability between the learning model and the surrogate oracle, proving its universality under mild assumptions. Experiments on stochastic vehicle scheduling demonstrate strong generalization performance. This work provides the first verifiable theoretical generalization framework for contextual stochastic optimization.

In reinforcement learning, reward modeling suffers from “error-regret mismatch”: low test error of the reward model does not guarantee low regret of the optimized policy, primarily due to distributional shift induced by policy optimization. Method: We provide the first theoretical proof that, for any arbitrarily small expected test error, there exist underlying data distributions yielding arbitrarily large regret. We construct explicit counterexamples, derive tight quantitative bounds linking reward estimation error and policy regret, and analyze the robustness of regularization techniques—including RLHF—against this mismatch. Contribution/Results: We show that low test error only ensures a worst-case regret upper bound, not actual policy performance; moreover, standard regularizers fail to eliminate the mismatch. Our analysis establishes a new theoretical benchmark for assessing reward model reliability and safety alignment in preference-based RL, with implications for trustworthy reward learning and deployment-critical applications.

This paper addresses the estimation error arising in machine learning when non-differentiable or hard-to-optimize target losses (e.g., 0–1 error) are replaced by surrogate losses (e.g., cross-entropy). We establish a consistency theory tailored to a given hypothesis class $H$. First, we introduce $H$-consistency bounds—strictly stronger than Bayes-consistency and $H$-calibration—and derive tight distribution-dependent and distribution-independent bounds for binary and multiclass settings. Second, we provide the first tight $H$-consistency bounds for max-, sum-, and comp-sum-type surrogates—including cross-entropy and MAE—as well as for adversarially robust learning. Third, we characterize the fundamental unattainability of perfect calibration and reveal a square-root growth law for the estimation error. Finally, we propose a smoothed adversarial surrogate loss and empirically validate its effectiveness. Our results deliver distribution-adaptive theoretical guarantees for non-Bayesian, non-differentiable learning tasks.

Existing generalization bounds in deep learning are often vacuous, incomputable, or restricted to specific model architectures. This work proposes a general and verifiable generalization bound by constructing a surrogate model that estimates the true risk through prediction disagreement on unlabeled data, without requiring any modification to the original model or training procedure. The approach unifies sample compression, model compression, and PAC-Bayes theory, yielding a framework applicable to arbitrary predictors and compatible with multiple theoretical paradigms. Empirical evaluations demonstrate that the resulting bounds are both tight and effective, providing reliable certificates of generalization performance across diverse models.

This work addresses the lack of formal performance guarantees for unseen tasks in multitask reinforcement learning within safety-critical settings. To this end, it presents the first generalization error bound applicable to arbitrary unknown task distributions. By integrating a lower bound on single-task performance—derived from finite trajectory data—with task-level generalization capacity, the paper establishes a high-confidence framework for policy performance assurance. The approach combines probabilistic generalization theory with confidence-bound analysis to deliver rigorous performance guarantees for novel tasks. Empirical validation across multiple state-of-the-art multitask RL algorithms demonstrates that the proposed bound is both theoretically sound and practically sample-efficient.

This work addresses the challenge of effectively transferring benchmark evaluation knowledge to novel few-shot tasks. It proposes a transfer learning and model selection aggregation method grounded in a weak monotonicity assumption: models that perform better across multiple source benchmarks tend to also perform better on the target task. The approach prunes the model class, models approximate weak monotonic relationships between tasks, and employs an adaptive hedging strategy over the performance frontier set to balance performance discrepancies among candidate models. Theoretical analysis demonstrates that this framework yields statistical gains under the weak monotonicity condition, and empirical results confirm its significant superiority over existing methods in few-shot scenarios.

This work addresses the challenge of achieving calibeating—simultaneously ensuring prediction calibration and low regret—under a broad family of proper loss functions, including α-Tsallis losses and log loss. By adopting a Bregman divergence perspective, we develop a unified framework that generalizes calibeating to arbitrary proper losses for the first time. Within this framework, we introduce a “Be The Regularized Leader” algorithm together with a novel regret identity. Our approach substantially weakens the dependence on dimensionality in U-calibration guarantees and yields logarithmic regret bounds for the Tsallis loss family, improving upon existing results. This provides a powerful new tool for balancing calibration and regret control in online learning settings.