Discriminative classification with generative features: bridging Naive Bayes and logistic regression

2025-11-30

AI Summary

Generative models (e.g., Naïve Bayes) lack discriminative optimization capability, while discriminative models (e.g., logistic regression) ignore generative structure. Method: We propose Smart Bayes—a hybrid classification framework that embeds the log-density-ratio (LDR), a data-driven generative feature, into a logistic regression–style discriminator. A spline-based nonparametric estimator models the one-dimensional LDR, automatically learning feature weights and relaxing Naïve Bayes’ restrictive independence and fixed-prior assumptions. Contribution/Results: Smart Bayes synergistically integrates generative interpretability with discriminative predictive accuracy. Extensive experiments on synthetic and real-world datasets demonstrate statistically significant improvements over both Naïve Bayes and standard logistic regression. The results validate the efficacy and generalizability of the density-ratio–driven generative-discriminative co-modeling paradigm.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce Smart Bayes, a new classification framework that bridges generative and discriminative modeling by integrating likelihood-ratio-based generative features into a logistic-regression-style discriminative classifier. From the generative perspective, Smart Bayes relaxes the fixed unit weights of Naive Bayes by allowing data-driven coefficients on density-ratio features. From a discriminative perspective, it constructs transformed inputs as marginal log-density ratios that explicitly quantify how much more likely each feature value is under one class than another, thereby providing predictors with stronger class separation than the raw covariates. To support this framework, we develop a spline-based estimator for univariate log-density ratios that is flexible, robust, and computationally efficient. Through extensive simulations and real-data studies, Smart Bayes often outperforms both logistic regression and Naive Bayes. Our results highlight the potential of hybrid approaches that exploit generative structure to enhance discriminative performance.

Problem

• Bridges generative and discriminative classification models
• Enhances class separation with likelihood-ratio-based features
• Develops a flexible estimator for univariate log-density ratios

Innovation

• Integrates likelihood-ratio features into logistic regression
• Uses spline-based estimator for log-density ratios
• Relaxes fixed weights of Naive Bayes with data-driven coefficients