Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning

2025-07-23

AI Summary

To address the high communication overhead and slow convergence of Decentralized Stochastic Gradient Descent (D-SGD) in resource-constrained networks, this paper proposes an information-entropy-based joint node-and-link scheduling framework. Our method introduces an entropy-driven importance metric to dynamically assign activation probabilities for nodes and selection probabilities for communication links, enabling only a critical subset of nodes and links per iteration while strictly adhering to per-round communication budget constraints. The framework requires no global topology knowledge, ensuring both theoretical interpretability and lightweight implementation. Experiments demonstrate that, under low communication budgets, our approach achieves significantly faster convergence than betweenness-centrality-based baselines and reduces communication overhead by up to 60%. Under higher budgets, it matches or surpasses baseline performance, with link scheduling effectiveness comparable to or exceeding that of MATCHA.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

This paper addresses decentralized stochastic gradient descent (D-SGD) over resource-constrained networks by introducing node-based and link-based scheduling strategies to enhance communication efficiency. In each iteration of the D-SGD algorithm, only a few disjoint subsets of nodes or links are randomly activated, subject to a given communication cost constraint. We propose a novel importance metric based on information entropy to determine node and link scheduling probabilities. We validate the effectiveness of our approach through extensive simulations, comparing it against state-of-the-art methods, including betweenness centrality (BC) for node scheduling and extit{MATCHA} for link scheduling. The results show that our method consistently outperforms the BC-based method in the node scheduling case, achieving faster convergence with up to 60% lower communication budgets. At higher communication budgets (above 60%), our method maintains comparable or superior performance. In the link scheduling case, our method delivers results that are superior to or on par with those of extit{MATCHA}.

Problem

• Enhancing communication efficiency in decentralized SGD
• Optimizing node and link scheduling with entropy
• Reducing communication costs while maintaining performance

Innovation

• Node and link-based scheduling for decentralized learning
• Information entropy metric for scheduling probabilities
• Outperforms BC and MATCHA in efficiency