Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters

2025-01-21

AI Summary

To address the prohibitive computational cost of training Gaussian Mixture Models (GMMs) on large-scale, high-dimensional data, this paper proposes a variational inference algorithm integrated with Mixture of Factor Analyzers (MFA). The method employs low-rank covariance parameterization and sublinear stochastic sampling, achieving—for the first time—per-iteration time complexity of *O(D)* in dimensionality *D* and *O(1)* in number of components *C*. Crucially, the total number of pairwise distance computations scales sublinearly with both sample size *N* and component count *C*. Empirically, the algorithm successfully trains a GMM with over 10 billion parameters on billion-scale image data, completing in approximately nine hours on a single CPU—more than ten times faster than the current state-of-the-art. This breakthrough significantly enhances the scalability and practical applicability of GMMs for ultra-large-scale problems.

Technology Category

• Machine Learning: Learning with Manifolds
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Gaussian Mixture Models (GMMs) range among the most frequently used machine learning models. However, training large, general GMMs becomes computationally prohibitive for datasets with many data points $N$ of high-dimensionality $D$. For GMMs with arbitrary covariances, we here derive a highly efficient variational approximation, which is integrated with mixtures of factor analyzers (MFAs). For GMMs with $C$ components, our proposed algorithm significantly reduces runtime complexity per iteration from $mathcal{O}(NCD^2)$ to a complexity scaling linearly with $D$ and remaining constant w.r.t. $C$. Numerical validation of this theoretical complexity reduction then shows the following: the distance evaluations required for the entire GMM optimization process scale sublinearly with $NC$. On large-scale benchmarks, this sublinearity results in speed-ups of an order-of-magnitude compared to the state-of-the-art. As a proof of concept, we train GMMs with over 10 billion parameters on about 100 million images, and observe training times of approximately nine hours on a single state-of-the-art CPU.

Problem

• Large-scale Data
• Gaussian Mixture Models (GMMs)
• Resource Efficiency

Innovation

• Gaussian Mixture Models
• Factor Analyzers
• High-Dimensional Data Processing