A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning

2026-04-25

AI Summary

This work addresses the highly non-convex optimization challenge encountered when training deep neural networks with the Softmax cross-entropy loss. To mitigate this issue, the authors propose a layer decoupling strategy that introduces auxiliary variables corresponding to hidden layer outputs, thereby decomposing the original deeply nested problem into a sequence of more tractable subproblems. An alternating minimization algorithm is developed to solve the resulting formulation. Theoretical analysis demonstrates that the proposed decoupled model is equivalent to the original loss, with its objective function serving as a tight upper bound and guaranteeing monotonic loss decrease under mild conditions. Experimental results confirm that the method substantially enhances training stability and convergence performance for both fully connected and convolutional neural networks.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks.

Problem

• cross-entropy loss
• deep learning optimization
• nonconvexity
• layer separation
• softmax

Innovation

• layer separation
• cross-entropy optimization
• nonconvex optimization
• alternating minimization
• deep learning