🤖 AI Summary
This work addresses the degeneracy issues in deep Linear Discriminant Analysis (LDA) under maximum likelihood training, which often leads to collapsed class means and covariances, thereby degrading discriminative performance. While cross-entropy training achieves high accuracy, it compromises the probabilistic coherence of the generative model. To reconcile these concerns, the paper introduces a Discriminative Negative Log-Likelihood (DNLL) loss that preserves LDA’s generative structure while incorporating a lightweight penalty on the mixture density to suppress overlap among high-probability regions across classes. This approach enables clear separation in the latent space and represents the first effective integration of generative modeling with discriminative training. Empirical results demonstrate that DNLL attains classification accuracy comparable to Softmax on both synthetic and standard image benchmarks, while significantly improving prediction calibration and feature discriminability.
📝 Abstract
We show that for unconstrained Deep Linear Discriminant Analysis (LDA) classifiers, maximum-likelihood training admits pathological solutions in which class means drift together, covariances collapse, and the learned representation becomes almost non-discriminative. Conversely, cross-entropy training yields excellent accuracy but decouples the head from the underlying generative model, leading to highly inconsistent parameter estimates. To reconcile generative structure with discriminative performance, we introduce the \emph{Discriminative Negative Log-Likelihood} (DNLL) loss, which augments the LDA log-likelihood with a simple penalty on the mixture density. DNLL can be interpreted as standard LDA NLL plus a term that explicitly discourages regions where several classes are simultaneously likely. Deep LDA trained with DNLL produces clean, well-separated latent spaces, matches the test accuracy of softmax classifiers on synthetic data and standard image benchmarks, and yields substantially better calibrated predictive probabilities, restoring a coherent probabilistic interpretation to deep discriminant models.