Muon Is Theoretically Wrong For Convolutions, But Empirically Effective

2026-10-05

AI Summary

This study addresses the theoretical breakdown of the Muon optimizer when applied to convolutional kernels, where reliance on tensor reshaping invalidates its underlying orthogonalization guarantees. To resolve this, we formally define the optimization objective under convolutional operator geometry for the first time and propose the Convolutional Newton-Schulz (Conv-NS) algorithm, which approximates the polar factor while preserving kernel support. We further investigate the over-constrained orthogonalization hypothesis. Experimental results demonstrate that although Conv-NS achieves computational efficiency comparable to standard implementations, it yields no significant accuracy improvements. This finding confirms that standard Muon remains practically effective for convolutional tasks despite its theoretical deficiencies in this context. By elucidating this theory-practice discrepancy and releasing our source code, this work provides critical insights and establishes a foundation for future optimizer development.

Abstract

Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon. To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schulz (Conv-NS), which approximates the polar factor in this geometry while preserving kernel support. When applied in fast training experiments, Conv-NS and reshape-based Muon are both computationally efficient and achieve comparable accuracy on CIFAR-10 and ImageNet classification tasks. However, as one could expect a theoretically aligned Conv-NS to outperform reshape-based Muon, we investigate this mismatch between practice and theoretical understanding, with the hypothesis that exact convolutional orthogonalization may overconstrain updates. These findings highlight Muon's strong practical performance while opening directions for its further development on convolutions. Our code is publicly available at \href{https://github.com/thib-s/muonconv-cifar10-airbench}{github conv-muon}.

Problem

• Muon optimizer
• convolutional kernels
• orthogonalization
• theory-practice gap

Innovation

• Muon optimizer
• Convolutional Newton-Schulz
• convolutional operator geometry
• polar factor approximation
• orthogonalization