On the best approximation by finite Gaussian mixtures

2023-06-25

International Symposium on Information Theory

AI Summary

This paper studies the approximation of general Gaussian location mixture models by finite Gaussian mixture models (GMMs), aiming to determine the minimal number of components required to achieve a prescribed accuracy under f-divergence. We propose a novel framework based on spectral analysis of a weighted moment matrix, establishing for the first time an exact equivalence between the optimal approximation error and the low-rank approximation error of this matrix. Leveraging local moment matching, characterization of the triangular moment structure, and refined estimation of the smallest eigenvalue, we derive tight upper and lower bounds—up to constant factors—for compactly supported or sub-Gaussian mixtures. Our results not only correct and substantially strengthen classical lower bounds for Gaussian mixtures but also achieve optimality in the leading constant factors for both bounds. This provides a rigorous theoretical foundation for complexity control and learnability analysis of GMMs.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We consider the problem of approximating a general Gaussian location mixture by finite mixtures. The minimum order of finite mixtures that achieve a prescribed accuracy (measured by various f-divergences) are determined within constant factors for the family of compactly supported or subgaussian mixing distributions. While the upper bound is achieved using the technique of local moment matching, the lower bound is established by relating the best approximation error to the low-rank approximation of certain trigonometric moment matrices and weighted moment matrices, followed by a refined spectral analysis of the minimum eigenvalue of these matrices. In the case of Gaussian mixing distributions, this result corrects a previous lower bound in [1].

Problem

• Approximating general Gaussian mixtures with finite mixtures
• Determining minimum order for prescribed approximation accuracy
• Correcting previous lower bound for Gaussian mixing distributions

Innovation

• Uses local moment matching technique
• Relates error to trigonometric matrices
• Refines spectral eigenvalue analysis