Cordial Learning: Distributed Training with Correlated Data

2026-10-02

AI Summary

This study addresses the performance degradation of decentralized methods caused by data correlation in distributed learning, as well as the privacy and communication bottlenecks inherent in centralized approaches. To overcome these challenges, this work proposes a framework termed "Affinity Learning," which facilitates collaborative training by exchanging only low-dimensional model outputs and extracting peer signals via local models, thereby avoiding raw data sharing. Theoretically, from a model-dependent game-theoretic perspective, it is proven that under linear assumptions, the non-convex global objective converges to the optimal solution with probability one. Experimental results demonstrate that the proposed framework significantly outperforms conventional federated learning on highly nonlinear tasks, such as structured multi-digit MNIST, effectively balancing privacy preservation with model performance.

Abstract

We consider a distributed learning task with agents that have correlated data. Specifically, the label of an agent depends on the input of other agents for the same sample, and these inputs are also correlated. Correlated data is the reality when agents share the same environment. Existing decentralized methods, such as federated learning, ignore the structure of the problem and perform poorly on correlated data. On the other hand, centralized approaches are infeasible due to privacy and communication constraints. We introduce cordial (correlated and distributed) learning to address this gap by sharing only low-dimensional outputs between the agents while training local models to extract informative signals from peers. This distributed learning induces a game in which the loss function of each agent depends on the models of others. Assuming a linear model, we prove that cordial learning converges with probability one to a globally optimal solution, despite the nonconvex global objective. Experiments on structured multi-digit MNIST tasks demonstrate that cordial learning remains highly effective even in highly nonlinear settings.

Problem

• distributed learning
• correlated data
• decentralized training
• federated learning
• privacy constraints

Innovation

• Distributed Learning
• Correlated Data
• Game Theory
• Nonconvex Optimization
• Federated Learning