Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets

2024-10-02

arXiv.org

AI Summary

This work addresses the global optimization challenge of two-layer quadratic-activation networks on abelian group reasoning tasks (e.g., modular addition). We propose CoGO, the first framework to reveal that the solution space admits a semiring algebraic structure and that the L2 loss decomposes into a sum of ring-homomorphic monomial potentials. Leveraging this structure, global optima can be analytically constructed via ring addition and multiplication from local optima. We theoretically prove that overparameterization asymptotically decouples training dynamics, substantially improving optimization tractability. Experiments demonstrate: (i) 95% of gradient descent solutions exactly match the theoretically derived global optima; (ii) global optimality is achieved even with small hidden-layer widths; and (iii) the theoretically predicted preference for low-complexity solutions effectively suppresses high-order, overfitting-prone solutions.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We prove rich algebraic structures of the solution space for 2-layer neural networks with quadratic activation and $L_2$ loss, trained on reasoning tasks in Abelian group (e.g., modular addition). Such a rich structure enables analytical construction of global optimal solutions from partial solutions that only satisfy part of the loss, despite its high nonlinearity. We coin the framework as CoGO (Composing Global Optimizers). Specifically, we show that the weight space over different numbers of hidden nodes of the 2-layer network is equipped with a semi-ring algebraic structure, and the loss function to be optimized consists of monomial potentials, which are ring homomorphism, allowing partial solutions to be composed into global ones by ring addition and multiplication. Our experiments show that around $95%$ of the solutions obtained by gradient descent match exactly our theoretical constructions. Although the global optimizers constructed only required a small number of hidden nodes, our analysis on gradient dynamics shows that over-parameterization asymptotically decouples training dynamics and is beneficial. We further show that training dynamics favors simpler solutions under weight decay, and thus high-order global optimizers such as perfect memorization are unfavorable. Code can be found at https://github.com/facebookresearch/luckmatters/tree/yuandong3/ssl/real-dataset.

Problem

• Analyzing algebraic structures in neural network solution spaces
• Constructing global optimal solutions from partial components
• Understanding gradient dynamics in overparameterized reasoning tasks

Innovation

• Analytical construction of global solutions from partial ones
• Semi-ring algebraic structure in weight space
• Sum potentials enable composition via ring operations