DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data

2026-06-02

AI Summary

This work addresses the privacy leakage inherent in gradient exchange during decentralized learning, particularly the challenge of simultaneously preserving privacy and maintaining model performance under non-IID data distributions. To this end, the paper proposes DPDL, a differentially private decentralized learning algorithm that integrates differential privacy into a decentralized stochastic optimization framework. DPDL perturbs inter-node gradients with calibrated Gaussian noise and innovatively combines cosine similarity–based gradient alignment with momentum-enhanced local updates to preserve model utility while ensuring privacy. Theoretical analysis establishes convergence conditions and derives the minimal noise magnitude required for privacy guarantees. Empirical evaluations demonstrate that DPDL effectively resists privacy attacks on real-world non-IID datasets, achieving high-accuracy training and near-linear speedup.

Abstract

In the paradigm of decentralized learning, a group of agents collaborate to train a global model using distributed datasets without a central server. Although the power of collaboration has been verified by many state-of-the-art studies, it entails extensive gradient information exchanging among the agents and thus induces high risk of privacy leakage for the individual agents. Moreover, in real-world applications, the training data are usually non-identically and independently distributed across the agents, inducing more challenges to enable privacy-preserved decentralized learning. To address these issues, we propose a privacy-preserved decentralized learning algorithm with non-IID data, DPDL, which leverages the notion of Differential Privacy (DP) in cross-gradient aggregation through a similarity-based calibration technique. Specifically, in each round, each agent perturbs the cross-gradients (i.e., the derivatives of its neighbors' local model in its private local data) by Gaussian noise mechanism before sharing them with its neighbors; it then adopt cosine similarity to calibrate the received perturbed cross-gradients such that the aggregation of the calibrated cross-gradients can be utilized to effectively update local model in a momentum-like manner. Our rigorous theoretical analysis not only reveals the minimum noise level required to achieve a specific level of privacy preservation, but also illustrates that our algorithm still achieves a linear speedup in training with non-IID data. We finally conduct extensive experiments on real-world dataset to validate the effectiveness of our algorithm in defending privacy attacks and in training accurate models.

Problem

• Differential Privacy
• Decentralized Learning
• Non-IID Data
• Privacy Preservation
• Gradient Exchange

Innovation

• Differential Privacy
• Decentralized Learning
• Non-IID Data
• Gradient Calibration
• Gaussian Noise Mechanism