Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables

2025-04-30

AI Summary

Gradient descent in least-squares learning of deep neural networks—including physics-informed neural networks (PINNs)—suffers from slow convergence and low accuracy due to the highly non-convex loss landscape and vanishing gradients. Method: We propose a hierarchical auxiliary-variable-based adaptive weighting optimization framework. By introducing layer-wise decoupled auxiliary variables and reformulating the loss function—guided by Lagrangian relaxation principles and an adaptive weight mechanism—we rigorously preserve equivalence to the original mean squared error under theoretical guarantees. Contribution/Results: This is the first method to enable adaptive adjustment of auxiliary variable weights, effectively mitigating inter-layer coupling and gradient degradation. Numerical experiments across diverse network architectures demonstrate that our approach accelerates convergence by 2–5× over standard gradient descent, while achieving higher final accuracy and improved robustness.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.

Problem

• Addresses inefficiency in deep learning gradient descent
• Introduces auxiliary variables to separate neural network layers
• Ensures consistency between reformulated and original loss functions

Innovation

• Introduces auxiliary variables for layer separation
• Uses self-adaptive weights for loss consistency
• Reformulates loss functions for easier optimization