A Robust Framework for Model Order Selection in Correlated Large-Dimensional CES Noise

2026-06-04

AI Summary

This work addresses the challenging problem of estimating the model order—i.e., the rank of the signal subspace—in the presence of high-dimensional, correlated, and non-Gaussian complex elliptically symmetric (CES) noise. The authors propose a two-stage robust framework: first, a Toeplitz-constrained M-estimator is employed to whiten the unknown covariance structure; second, the signal subspace rank is inferred using large-dimensional random matrix theory (RMT). This approach uniquely integrates Toeplitz-rectified M-estimators—including the sample covariance matrix (SCM), Maronna’s, and Tyler’s estimators—with large-dimensional RMT to construct an almost surely consistent order estimator and derive an explicit eigenvalue separation threshold. Experiments on synthetic data as well as real-world hyperspectral images, electroencephalographic, and financial datasets demonstrate that the proposed method significantly outperforms conventional criteria such as AIC, confirming its robustness and effectiveness.

Abstract

This paper addresses model order selection under large-dimensional, correlated, non-Gaussian noise. Sources are assumed to be embedded in additive Complex Elliptically Symmetric (CES) noise with an unknown Toeplitz-structured scatter matrix. We propose a two-stage robust framework: (i) a noise-whitening step based on a Toeplitz-rectified $M$-estimator of the scatter matrix, and (ii) signal subspace rank inference via large-dimensional Random Matrix Theory (RMT). Almost sure consistency of the proposed estimators is established, together with explicit RMT eigenvalue upper bounds separating signal from noise components, in the regime where the observation dimension $m$ and the sample size $N$ grow proportionally. Three estimation branches are derived, based respectively on the sample covariance matrix (SCM), Maronna's $M$-estimator, and the distribution-free Tyler $M$-estimator for whitening. The methodology is validated on synthetic data, real hyperspectral images, EEG recordings, and financial data, with significant gains over AIC and unwhitened methods.

Problem

• model order selection
• large-dimensional noise
• Complex Elliptically Symmetric noise
• correlated noise
• non-Gaussian noise

Innovation

• Model Order Selection
• Complex Elliptically Symmetric Noise
• Toeplitz-structured Scatter Matrix
• Random Matrix Theory
• M-estimator