When Does Muon Help Agentic Reinforcement Learning?

2026-07-17

AI Summary

This work investigates the efficacy and applicability of the Muon optimizer in post-training sparse-reward reinforcement learning. Using the ALFWorld environment with a Qwen2.5-0.5B-Instruct model and Group-in-Group policy optimization variants (GiGPO, GRPO, GraphGPO), the study conducts single-seed controlled experiments to systematically compare Muon against AdamW. It reveals, for the first time, that under specific learning rate and advantage estimator configurations, Muon substantially improves agent success rates—for instance, achieving a success rate of 0.901 with GraphGPO at a learning rate of 1e-5, elevating validation AUC from 0.399 to 0.556—and accelerates convergence. These findings highlight a promising new direction in the co-design of optimizers, advantage estimation, and learning rate schedules.

Abstract

Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.

Problem

• Muon
• agentic reinforcement learning
• sparse-reward
• policy optimizer
• advantage estimator

Innovation

• Muon optimizer
• agentic reinforcement learning
• sparse-reward RL
• policy optimization
• advantage estimation