Communication-Efficient Split Learning via Adaptive Feature-Wise Compression

2023-07-20

arXiv.org

AI Summary

To address the excessive communication overhead of intermediate features and gradients in Split Learning (SL) for edge-based collaborative training, this paper proposes SplitFC—a novel framework introducing the first dual-strategy adaptive compression mechanism based on column-wise vector dispersion. Specifically: (1) a standard-deviation-driven feature-level Dropout dynamically prunes low-informativeness channels; and (2) a range-driven adaptive quantization scheme, solved in closed form for optimal quantization levels, minimizes reconstruction error. Additionally, chained gradient clipping ensures convergence stability. Extensive experiments on MNIST, CIFAR-100, and CelebA demonstrate that SplitFC reduces communication volume by up to 87% while maintaining accuracy comparable to state-of-the-art SL methods. The framework establishes a new paradigm for efficient, communication-aware collaborative training at the edge.

Abstract

This paper proposes a novel communication-efficient split learning (SL) framework, named SplitFC, which reduces the communication overhead required for transmitting intermediate feature and gradient vectors during the SL training process. The key idea of SplitFC is to leverage different dispersion degrees exhibited in the columns of the matrices. SplitFC incorporates two compression strategies: (i) adaptive feature-wise dropout and (ii) adaptive feature-wise quantization. In the first strategy, the intermediate feature vectors are dropped with adaptive dropout probabilities determined based on the standard deviation of these vectors. Then, by the chain rule, the intermediate gradient vectors associated with the dropped feature vectors are also dropped. In the second strategy, the non-dropped intermediate feature and gradient vectors are quantized using adaptive quantization levels determined based on the ranges of the vectors. To minimize the quantization error, the optimal quantization levels of this strategy are derived in a closed-form expression. Simulation results on the MNIST, CIFAR-100, and CelebA datasets demonstrate that SplitFC outperforms state-of-the-art SL frameworks by significantly reducing communication overheads while maintaining high accuracy.

Problem

• Reduces communication overhead in split learning
• Compresses intermediate feature and gradient vectors
• Maintains high accuracy while minimizing data transmission

Innovation

• Adaptive feature-wise dropout strategy
• Adaptive feature-wise quantization method
• Closed-form optimal quantization levels derivation