Partitioned Neural Network Training via Synthetic Intermediate Labels

2024-03-17

arXiv.org

AI Summary

To address GPU memory constraints and inter-device communication bottlenecks in distributed training of large neural networks, this paper proposes a segmented collaborative training framework based on synthetic labels: lightweight synthetic intermediate labels replace high-dimensional real feature transmissions, decoupling forward and backward propagation dependencies. The framework introduces a label generator, consistency regularization, and segmented asynchronous backpropagation to eliminate gradient synchronization overhead. Evaluated on ResNet-50/CIFAR-10, the method reduces communication volume by 92%, accelerates training by 2.3×, incurs <0.5% accuracy degradation, and scales losslessly to an 8-GPU cluster. This work is the first to systematically apply synthetic labels to cross-device collaborative training, significantly improving training efficiency and scalability for large-scale models.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Segmentation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Problem

• Reduces GPU memory usage
• Minimizes data communication overhead
• Maintains model accuracy efficiently

Innovation

• Partitioned neural network training
• Synthetic intermediate labels
• Reduced memory and computational load