🤖 AI Summary
This work investigates whether higher-order adaptive Runge–Kutta (RK) optimizers genuinely improve neural network training and generalization under computationally matched conditions. We construct an Adam variant based on the Bogacki–Shampine 3(2) RK pair (RK-Adam) and conduct preregistered comparative experiments under a strict gradient computation budget. Our findings reveal that RK-Adam’s “adaptive” step size is effectively fixed, and its gradient averaging mechanism introduces an implicit regularization effect. Across all ten random seeds, RK-Adam outperforms learning-rate-matched Adam and AdamW but underperforms RMSprop and NAdam. Although it achieves approximately 40× lower training loss in full-batch settings, this does not translate into improved test accuracy. The study clarifies the actual behavior of RK-based optimizers and identifies the source of their limited generalization benefits.
📝 Abstract
Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.