Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks

2026-04-27

AI Summary

This work proposes Self-Abstracted Learning (SAL), a novel framework designed to address key challenges in deep neural network training, including vanishing gradients, overfitting, and optimization instability. SAL employs a top-down, hierarchical training strategy that begins with a simple network at the highest layer and progressively guides the sequential optimization of deeper, more complex subnetworks using the hidden representations and outputs of already-trained upper layers as supervisory signals. By introducing a hierarchical self-guidance mechanism, SAL consistently outperforms conventional end-to-end training across diverse architectures—including multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs)—particularly in data-scarce settings. The approach demonstrates markedly improved generalization capability and training stability while maintaining architectural flexibility.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Training large-scale deep neural networks effectively and stably is essential for applying deep learning across various fields. However, conventional methods, which rely on training a single large network, often encounter challenges such as gradient vanishing, overfitting and unstable learning. To overcome these limitations, we introduce Self-Abstraction Learning (SAL), a hierarchical framework. In SAL, networks are arranged by structural complexity, where the simplest topmost network is trained first and its hidden and output layers serve as guidance for the successively more complex networks below. This top-down sequential guidance effectively mitigates optimization issues, enabling stable training of deep architectures. Various experiments across MLP, CNN, and RNN architectures demonstrate that SAL consistently outperforms conventional methods, ensuring robust generalization even in data-scarce and complex network regimes.

Problem

• deep neural networks
• stable training
• gradient vanishing
• overfitting
• optimization issues

Innovation

• Self-Abstraction Learning
• hierarchical training
• stable deep learning
• top-down guidance
• generalization