🤖 AI Summary
This work explores the use of fixed-depth symbolic regression to automatically discover neural network optimizers that outperform hand-designed counterparts. The method systematically searches within an expression space composed of gradients, momentum, and adaptive terms to identify compact and efficient weight update rules. It introduces a novel mechanism for constructing nonlinear rational expressions and represents the first application of fixed-depth symbolic regression to optimizer discovery. Experimental results across 30 benchmark–architecture combinations demonstrate that the discovered update rules surpass extensively tuned state-of-the-art optimizers in 25 cases, achieving an average reduction of 44.47% in mean squared error. These findings reveal both structural commonalities and diversity among highly effective update rules.
📝 Abstract
We investigate whether symbolic regression can discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules are represented as fixed-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44.47\% over the improved cases. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger-scale validation.