Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

2026-06-02

AI Summary

This work uncovers the theoretical essence of momentum in the Muon optimizer by formulating a signal–noise decomposition of gradients. It demonstrates that, in matrix optimization, momentum acts as a spectral filter that preferentially suppresses gradient noise while preserving dominant signal components, followed by an orthogonalization-based update. This “denoise-then-orthogonalize” strategy effectively widens the spectral gap between signal and noise, thereby stabilizing the singular subspace and enhancing both update stability and directional alignment. The study provides the first spectral filtering interpretation of momentum in matrix optimizers, offering a theoretical foundation for its empirical success in large language model training. Experimental validation on pretraining tasks confirms the efficacy of this mechanism.

Abstract

Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers.

Problem

• momentum
• Muon
• spectral filtering
• matrix-based optimizers
• gradient signal

Innovation

• spectral filtering
• momentum
• orthogonalization
• singular subspaces
• matrix-based optimization