Preconditioned subgradient method for composite optimization: overparameterization and fast convergence

2025-09-14

AI Summary

For composite optimization problems where ill-conditioning or over-parameterization of the smooth mapping impedes subgradient methods to sublinear convergence, this paper proposes a preconditioned Levenberg–Marquardt-type subgradient method. Unlike conventional approaches, it does not require the smooth mapping to be well-conditioned; instead, it only assumes standard regularity conditions on the convex component, thereby achieving global linear convergence—even in over-parameterized regimes where traditional methods suffer from convergence degradation. The algorithm integrates preconditioning with adaptive regularization, fully exploiting the composite structure of the objective. Theoretical analysis establishes its applicability to canonical nonconvex problems, including phase retrieval, matrix sensing, and tensor decomposition. Numerical experiments demonstrate significantly accelerated convergence and enhanced robustness.

Technology Category

Search and Optimization: Non-convex OptimizationMachine Learning: OptimizationComputer Vision: Learning & Optimization for CV

Application Category

Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingWeb Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Composite optimization problems involve minimizing the composition of a smooth map with a convex function. Such objectives arise in numerous data science and signal processing applications, including phase retrieval, blind deconvolution, and collaborative filtering. The subgradient method achieves local linear convergence when the composite loss is well-conditioned. However, if the smooth map is, in a certain sense, ill-conditioned or overparameterized, the subgradient method exhibits much slower sublinear convergence even when the convex function is well-conditioned. To overcome this limitation, we introduce a Levenberg-Morrison-Marquardt subgradient method that converges linearly under mild regularity conditions at a rate determined solely by the convex function. Further, we demonstrate that these regularity conditions hold for several problems of practical interest, including square-variable formulations, matrix sensing, and tensor factorization. Numerical experiments illustrate the benefits of our method.

Problem

• Optimizing composite functions with ill-conditioned smooth maps
• Addressing slow sublinear convergence in overparameterized problems
• Improving convergence rates for data science applications

Innovation

• Levenberg-Morrison-Marquardt subgradient method
• Linear convergence under mild conditions
• Applied to matrix sensing and tensor factorization