Simplex Deep Linear Discriminant Analysis

2026-01-04

AI Summary

This work proposes a geometrically constrained formulation of Deep Linear Discriminant Analysis (Deep LDA) to address performance degradation caused by class cluster overlap or collapse during end-to-end maximum likelihood training. By fixing the class means in the latent space to the vertices of a regular simplex and assuming a shared spherical covariance, the method eliminates degenerate solutions while preserving model simplicity and interpretability. This design enables stable maximum likelihood optimization and yields well-separated class representations. Experimental results on Fashion-MNIST, CIFAR-10, and CIFAR-100 demonstrate that the approach achieves classification accuracy comparable to Softmax baselines, while its latent embeddings exhibit highly structured geometric arrangements in two-dimensional projections.

Technology Category

Machine Learning: Learning with ManifoldsComputer Vision: Representation Learning for VisionNatural Language Processing: Learning & Optimization for NLP

Application Category

Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphsSemantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semanticsEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of how to train the resulting deep classifier by maximum likelihood estimation (MLE). We first show that end-to-end MLE training of an unconstrained Deep LDA model ignores discrimination: when both the LDA parameters and the encoder parameters are learned jointly, the likelihood admits a degenerate solution in which some of the class clusters may heavily overlap or even collapse, and classification performance deteriorates. Batchwise moment re-estimation of the LDA parameters does not remove this failure mode. We then propose a constrained Deep LDA formulation that fixes the class means to the vertices of a regular simplex in the latent space and restricts the shared covariance to be spherical, leaving only the priors and a single variance parameter to be learned along with the encoder. Under these geometric constraints, MLE becomes stable and yields well-separated class clusters in the latent space. On images (Fashion-MNIST, CIFAR-10, CIFAR-100), the resulting Deep LDA models achieve accuracy competitive with softmax baselines while offering a simple, interpretable latent geometry that is clearly visible in two-dimensional projections.

Problem

• Deep Linear Discriminant Analysis
• maximum likelihood estimation
• class collapse
• degenerate solution
• latent space

Innovation

• Deep Linear Discriminant Analysis
• maximum likelihood estimation
• regular simplex
• spherical covariance
• constrained latent geometry