Improving Learning to Optimize Using Parameter Symmetries

2025-04-21

AI Summary

This work addresses the under-modeling of parameter-space symmetries in learned optimization (L2O). We propose a symmetry-aware L2O framework that jointly learns symmetry transformations—induced by group actions—and local update rules. Theoretically, we establish, for the first time, that our method locally approximates Newton’s method even when the optimal group element is unknown, and we constructively provide rigorous examples of learnable symmetry transformations. Methodologically, we integrate group-theoretic modeling, meta-learning, and optimization dynamics analysis, introducing a momentum-augmented “teleportation” mechanism to improve generalization. Evaluated on a newly constructed symmetry-aware benchmark, our algorithm achieves significant gains in convergence speed and cross-task generalization. Failure-case analysis further characterizes the precise conditions under which symmetry modeling remains effective.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization.

Problem

• Exploiting parameter symmetry to enhance optimization efficiency
• Learning symmetry transformations improves meta-optimizer performance
• Leveraging neural network symmetry for better meta-optimization

Innovation

• Exploits parameter space symmetry for optimization
• Learns symmetry transformations and local updates
• Enhances performance with momentum techniques