robust em covariance estimation

This work addresses the challenge of robust covariance matrix estimation under the scale-invariant random vector (SIRV) model in the presence of ignorable missing data. The authors propose a novel expectation-maximization (EM) algorithm that introduces an inverse-Gamma prior on the scale variables, thereby recasting the observation model as a complex multivariate Student-t distribution. This formulation enables, for the first time within the SIRV framework, closed-form updates in both the E-step and M-step. The method further incorporates numerical optimizations—including Hermitian positive semi-definite constraints, regularized matrix inversion, and computational reuse—to significantly enhance numerical stability and efficiency. Experimental results demonstrate that the proposed approach effectively reconstructs missing entries and achieves superior denoising performance under both missing completely at random (MCAR) and missing not at random (MNAR) mechanisms, making it well-suited for processing Sentinel-1 InSAR time-series data.

This paper addresses linear regression under missing-at-random (MAR) data by proposing an ANCOVA-equivalent reformulation of the EM algorithm that simplifies computation. The method recasts EM iterations as standard linear or nonlinear regression procedures, enabling constrained prediction and asymptotic variance estimation. Through rigorous theoretical derivation, we establish six theorems that— for the first time—unify the maximum likelihood estimation (MLE) consistency foundations of diverse imputation strategies, thereby enhancing interpretability and implementation flexibility. The approach is validated within the SAS PROC MI framework and applied to reanalyze 14 canonical datasets; imputation results match the gold-standard reference exactly, confirming its accuracy, robustness, and broad applicability.

Standard Expectation-Maximization (EM) algorithms are ill-suited for statistical models with discrete latent variables due to their reliance on differentiability and continuous latent spaces. Method: We propose the first unified framework integrating Mirror Descent (MD) with Sequential Monte Carlo (SMC) for joint parameter estimation and posterior inference. Our approach jointly minimizes a variational functional over both the parameter space and the space of probability measures, enabling maximum likelihood estimation (MLE) without requiring latent variable continuity. Contribution/Results: This work breaks EM’s dependence on latent variable continuity, marking the first application of MD to MLE in discrete latent variable models, with rigorous convergence guarantees established. Experiments demonstrate significant improvements over standard EM across multiple discrete latent variable tasks; on real-valued latent variable benchmarks, our method matches state-of-the-art performance, validating both theoretical soundness and empirical robustness.

To address the challenge of scarce clean training data in Bayesian inverse problems, this paper proposes the first framework for learning theoretically grounded diffusion priors solely from incomplete and noisy observations. Methodologically, we embed diffusion probabilistic modeling into an Expectation-Maximization (EM) algorithm: the E-step estimates the latent variable posterior via iterative denoising sampling, while the M-step updates diffusion model parameters by maximizing the marginal likelihood. Our key contributions are: (1) the first provably consistent learning of diffusion priors directly from noisy and/or missing observations; and (2) an unconditional posterior sampling strategy that eliminates reliance on assumptions about the forward process. Experiments demonstrate that the learned prior achieves performance on par with fully supervised models in downstream inverse tasks—including denoising and inpainting—while ensuring rigorous generative consistency and theoretical guarantees.

Existing spectral shrinkage methods for high-dimensional covariance estimation rely either on heuristic objectives or strong distributional assumptions. To address this, we propose a geometrically unified framework grounded in distributionally robust optimization (DRO). Our approach defines geometric divergence measures—including KL, Fisher–Rao, and Wasserstein divergences—on the covariance manifold, constructs data-driven ambiguity sets, and automatically derives spectral shrinkage estimators by minimizing the worst-case Frobenius error. Crucially, we establish for the first time that the functional form of shrinkage is intrinsically determined by the geometric structure of the chosen divergence, thereby eliminating dependence on prior distributional assumptions. We prove asymptotic consistency of the proposed estimator and derive an $O(1/n)$ finite-sample error bound. Moreover, we obtain three explicit robust estimators, which match state-of-the-art performance on both synthetic and real-world datasets.

本文提出了一种高效的EM算法，用于处理矩阵正态混合模型中的元素级和结构缺失问题，通过坐标近似更新条件均值和协方差，显著降低了计算成本。

This work addresses the limitations of maximum likelihood estimation (MLE) in sketching algorithms, which often suffers from numerical instability and computational inefficiency, and clarifies its previously unclear theoretical relationship with control variate estimation (CVE). Under exponential family distributions, the paper establishes—for the first time—the asymptotic equivalence between MLE and the optimal CVE by proving they share the same asymptotic variance. Building on this theoretical insight, the authors propose a novel MLE framework based on the expectation–maximization (EM) algorithm. This approach substantially improves numerical stability and computational efficiency. Empirical evaluations on bivariate normal distributions demonstrate its superiority over conventional root-finding methods, thereby validating the theoretical findings and highlighting its practical utility.

This work addresses the limitations of the traditional Expectation–Maximization (EM) algorithm, which relies on ad hoc latent variable constructions and is restricted to specific missing-data settings, lacking a unified framework. The authors propose a Normalized EM (N-EM) algorithm that generalizes EM to log-likelihood optimization problems involving integral terms by introducing a normalized density function. This approach establishes a three-stage iterative scheme comprising a Normalization step (N-step), an Expectation step (E-step), and a Maximization step (M-step). For the first time, it provides a unified optimization framework applicable to a broader class of likelihood functions, eliminating the need for manually specified latent variables. The method not only solves problems intractable to conventional EM but also achieves efficient and consistent optimization in comparable scenarios. Theoretical analysis and extensive experiments confirm the convergence and effectiveness of the proposed algorithm.

This study clarifies the fundamental distinction between restricted maximum likelihood (REML) and maximum likelihood (ML) estimation in linear mixed models. Within the EM algorithm framework, the two methods differ solely in their treatment of the covariance matrix during variance component updates: REML employs the prediction error covariance—corresponding to Henderson’s C matrix—whereas ML uses the conditional covariance. This work is the first to explicitly interpret REML’s computational rationale through the lens of prediction error covariance and provides concise R code to transparently illustrate the key matrices involved. The implemented algorithm successfully reproduces both ML and REML results from the lme4 package, clearly exposing the core difference in their covariance structures and offering a reproducible, pedagogically valuable tool for understanding and teaching these estimation methods.

This study addresses the existence, uniqueness, and algorithmic convergence of covariance matrix estimators in penalized multivariate divergence-based M-estimation under structural constraints. By constructing a regularized M-estimation framework with geodesically convex (not necessarily smooth) penalty functions, the authors rigorously establish the well-posedness of the resulting estimator. They further demonstrate, for the first time, that the standard fixed-point algorithm may fail in this setting and propose a novel reweighted algorithm that integrates tools from geodesic convex analysis and nonsmooth optimization. This new method guarantees monotone convergence for a broad class of penalized M-estimation problems involving structured covariance matrices.