Convergence of Spectral Descent for Non-smooth Optimization

2026-05-26

AI Summary

This work addresses the lack of theoretical convergence guarantees for the Muon optimizer in nonsmooth optimization by focusing on its simplified variants—spectral descent (SD) and truncated spectral descent (TSD). Under standard assumptions of convexity, Lipschitz continuity, and sharpness, we establish the first global linear convergence guarantees for these methods and extend the analysis to their regularized counterparts with decoupled weight decay. We uncover an intrinsic connection between SD and the Frank–Wolfe algorithm, derive a sublinear convergence rate for regularized TSD, and provide rigorous recovery guarantees for robust low-rank matrix recovery. Experimental results corroborate the theoretical findings, confirming the efficacy and correctness of our analysis.

Abstract

The Muon optimizer has recently demonstrated remarkable empirical success in training large language models. However, the theoretical understanding of its mechanisms remains limited. Current convergence guarantees for Muon rely heavily on smoothness assumptions, leaving its non-smooth convergence behavior largely unexplored. In this work, we take a step toward bridging this gap by investigating Spectral Descent (SD), a simplified variant of Muon, together with its truncated counterpart, Truncated Spectral Descent (TSD). Under convexity, Lipschitz continuity, and sharpness conditions, we establish global linear convergence for both SD and TSD in non-smooth convex formulations. We also study regularized variants equipped with decoupled weight decay and derive sublinear convergence guarantees through their connection with Frank-Wolfe methods. Finally, we apply our theoretical framework to robust low-rank matrix recovery under mixed sparse and dense noise regimes and provide rigorous recovery guarantees. Numerical experiments support the theoretical findings and demonstrate the effectiveness of Muon-type methods for non-smooth optimization.

Problem

• non-smooth optimization
• convergence
• Muon optimizer
• Spectral Descent
• sharpness

Innovation

• Spectral Descent
• non-smooth optimization
• linear convergence
• Muon optimizer
• robust matrix recovery