Can Learning Be Explained By Local Optimality In Robust Low-rank Matrix Recovery?

2023-02-21

AI Summary

This paper investigates the optimization geometry of robust low-rank matrix recovery under ℓ₁ loss: specifically, whether the true rank-𝑟 matrix 𝑋* is a local minimum of the ℓ₁-loss objective under linear measurements corrupted by outliers. Method: Leveraging nonsmooth optimization analysis, critical point classification theory, curvature verification, and subgradient dynamics modeling on the low-rank manifold, the authors rigorously characterize the geometric nature of 𝑋*. Contribution/Results: Under mild assumptions, the authors establish for the first time that 𝑋* is not a local minimum but a strict saddle point—i.e., it admits a direction of negative curvature. This challenges the conventional wisdom that all saddle points must be avoided. Crucially, they show that subgradient descent can still recover 𝑋* successfully—not because 𝑋* is locally optimal, but due to the specific manifold structure at the saddle and the algorithm’s inherent escape dynamics. These findings provide novel theoretical foundations for robust learning.

Abstract

We explore the local landscape of low-rank matrix recovery, focusing on reconstructing a $d_1 imes d_2$ matrix $X^star$ with rank $r$ from $m$ linear measurements, some potentially noisy. When the noise is distributed according to an outlier model, minimizing a nonsmooth $ell_1$-loss with a simple sub-gradient method can often perfectly recover the ground truth matrix $X^star$. Given this, a natural question is what optimization property (if any) enables such learning behavior. The most plausible answer is that the ground truth $X^star$ manifests as a local optimum of the loss function. In this paper, we provide a strong negative answer to this question, showing that, under moderate assumptions, the true solutions corresponding to $X^star$ do not emerge as local optima, but rather as strict saddle points -- critical points with strictly negative curvature in at least one direction. Our findings challenge the conventional belief that all strict saddle points are undesirable and should be avoided.

Problem

• Explores local landscape of low-rank matrix recovery
• Investigates if ground truth matrix is local optimum
• Challenges belief strict saddle points are undesirable

Innovation

• Uses nonsmooth $ell_1$-loss for matrix recovery
• Employs simple sub-gradient optimization method
• Identifies strict saddle points, not local optima