WeSpeR: Population spectrum retrieval and spectral density estimation of weighted sample covariance

📅 2024-10-18
🏛️ arXiv.org
📈 Citations: 0
✨ Influential: 0
📄 PDF
🤖 AI Summary
This paper addresses the challenge of estimating the spectral distribution of high-dimensional weighted sample covariance matrices. We propose WeSpeR, a novel algorithm that (i) rigorously establishes, for the first time, that the limiting spectral distribution admits a regular continuous density; (ii) introduces the first end-to-end joint framework simultaneously performing continuous spectral density estimation, precise support set bounding, and population-level spectral inversion; and (iii) integrates asymptotic random matrix theory analysis, numerical Stieltjes transform inversion, adaptive support localization, and grid optimization. Experiments demonstrate that WeSpeR significantly improves spectral density estimation accuracy, faithfully recovers the true population eigenvalue distribution, and exhibits both theoretical convergence guarantees and strong robustness against model misspecification and noise. By unifying theoretical analysis and practical estimation, WeSpeR establishes a new paradigm for high-dimensional weighted covariance modeling.

Technology Category

Machine Learning: Matrix & Tensor MethodsIntelligent Robots: State EstimationReasoning under Uncertainty: Relational Probabilistic Models

Application Category

Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsResponsible Web: Human-perceived consequences of algorithmic deployment on the webSystems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
📝 Abstract
The spectrum of the weighted sample covariance shows a asymptotic non random behavior when the dimension grows with the number of samples. In this setting, we prove that the asymptotic spectral distribution $F$ of the weighted sample covariance has a continuous density on $mathbb{R}^*$. We address then the practical problem of numerically finding this density. We propose a procedure to compute it, to determine the support of $F$ and define an efficient grid on it. We use this procedure to design the $ extit{WeSpeR}$ algorithm, which estimates the spectral density and retrieves the true spectral covariance spectrum. Empirical tests confirm the good properties of the $ extit{WeSpeR}$ algorithm.
Problem

Research questions and friction points this paper is trying to address.

Computing non-linear shrinkage for weighted covariance
Speeding up high-dimensional covariance estimation
Deriving efficient algorithm for large-scale data
Innovation

Methods, ideas, or system contributions that make the work stand out.

WeSpeR algorithm speeds up non-linear shrinkage
Uses asymptotic sample spectrum theoretical properties
PyTorch implementation for high-dimensional covariance computation
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
Université Paris-Dauphine | Société Générale Corporate and Investment Banking
B
Benoit Oriol
CEREMADE, Université Paris-Dauphine, PSL, Paris, France Société Générale Corporate and Investment Banking, Puteaux, France