Risk-based Calibration for Generative Classifiers

2024-09-05

AI Summary

Generative classifiers traditionally rely on joint probability modeling, whose training objectives—such as maximum likelihood estimation—are fundamentally misaligned with the core supervised learning metric: the 0–1 loss (classification error rate). To address this gap, we propose a risk-calibrated iterative training framework that, for the first time, directly incorporates the 0–1 loss into parameter updates of generative models—including Naïve Bayes and Quadratic Discriminant Analysis. Our method leverages sample-level risk feedback to dynamically reweight sufficient statistics during estimation, thereby enhancing discriminative features for the true class and suppressing confounding signals from incorrect classes. This approach bridges the theoretical disconnect between generative modeling principles and discriminative performance optimization. Empirically, it consistently reduces both training and generalization errors across 20 heterogeneous benchmark datasets, outperforming standard closed-form solutions in all cases.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Generative classifiers are constructed on the basis of a joint probability distribution and are typically learned using closed-form procedures that rely on data statistics and maximize scores related to data fitting. However, these scores are not directly linked to supervised classification metrics such as the error, i.e., the expected 0-1 loss. To address this limitation, we propose a learning procedure called risk-based calibration (RC) that iteratively refines the generative classifier by adjusting its joint probability distribution according to the 0-1 loss in training samples. This is achieved by reinforcing data statistics associated with the true classes while weakening those of incorrect classes. As a result, the classifier progressively assigns higher probability to the correct labels, improving its training error. Results on 20 heterogeneous datasets using both na""ive Bayes and quadratic discriminant analysis show that RC significantly outperforms closed-form learning procedures in terms of both training error and generalization error. In this way, RC bridges the gap between traditional generative approaches and learning procedures guided by performance measures, ensuring a closer alignment with supervised classification objectives.

Problem

• Improves generative classifiers using 0-1 loss calibration
• Bridges gap between data fitting and classification metrics
• Enhances training and generalization error in supervised tasks

Innovation

• Risk-based calibration adjusts joint probability distribution
• Reinforces true class statistics, weakens incorrect ones
• Improves training and generalization error significantly