Hyperball May Not Be a Free Lunch

2026-07-24

AI Summary

This work investigates the source of the performance advantage in Hyperball-type optimizers, disentangling whether it stems from their update direction or learning rate scheduling. To this end, the authors introduce the notion of “angular effective learning rate” and decompose parameter updates into radial and tangential components, showing that conventional norm-based analyses correspond merely to a special case under orthogonal updates. Through angular displacement analysis, component decoupling, and large-scale pretraining experiments, they demonstrate that the superiority of Hyperball primarily arises from the dynamic evolution of its effective step size rather than an intrinsic update direction. Radial updates exert minimal influence on angular changes, whereas learning rate decay strategies profoundly affect convergence—accelerating early training but potentially impairing later-stage performance—highlighting the critical importance of fine-grained learning rate scheduling.

Abstract

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Problem

• Hyperball optimizer
• scale-invariant networks
• effective learning rate
• angular displacement
• learning-rate scheduling

Innovation

• angular effective learning rate
• Hyperball optimizer
• radial-tangential decomposition
• learning rate scheduling
• scale-invariant networks