LocalKMeans: Convergence of Lloyd's Algorithm with Distributed Local Iterations

2025-05-23

AI Summary

This paper investigates the convergence of distributed K-means in a multi-machine setting with local data: nodes perform Lloyd’s algorithm in parallel and synchronize centroids only every $L$ iterations. Addressing the non-convex, non-smooth, and latent-variable nature of the clustering objective, we establish the first local iteration convergence theory for LocalKMeans. Leveraging virtual iteration analysis, tight statistical characterization of Lloyd steps, and distributional robustness analysis under Gaussian mixture models, we prove linear convergence to a neighborhood of the global optimum while precisely quantifying the trade-off between local computation cost and the elevated signal-to-noise ratio threshold required for convergence. Our core contribution breaks a longstanding bottleneck in distributed unsupervised learning analysis by revealing, for the first time, the *modelable impact* of local iterations on clustering accuracy—thereby providing a rigorous theoretical foundation for the communication–accuracy trade-off in distributed clustering.

Technology Category

• Machine Learning: Clustering
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Security and Privacy: Large-scale security measurements

Abstract

In this paper, we analyze the classical $K$-means alternating-minimization algorithm, also known as Lloyd's algorithm (Lloyd, 1956), for a mixture of Gaussians in a data-distributed setting that incorporates local iteration steps. Assuming unlabeled data distributed across multiple machines, we propose an algorithm, LocalKMeans, that performs Lloyd's algorithm in parallel in the machines by running its iterations on local data, synchronizing only every $L$ of such local steps. We characterize the cost of these local iterations against the non-distributed setting, and show that the price paid for the local steps is a higher required signal-to-noise ratio. While local iterations were theoretically studied in the past for gradient-based learning methods, the analysis of unsupervised learning methods is more involved owing to the presence of latent variables, e.g. cluster identities, than that of an iterative gradient-based algorithm. To obtain our results, we adapt a virtual iterate method to work with a non-convex, non-smooth objective function, in conjunction with a tight statistical analysis of Lloyd steps.

Problem

• Analyzing convergence of distributed Lloyd's algorithm with local iterations
• Studying trade-off between local steps and required signal-to-noise ratio
• Adapting virtual iterate method for non-convex unsupervised learning

Innovation

• Distributed local iterations for Lloyd's algorithm
• Synchronization every L steps in LocalKMeans
• Virtual iterate method for non-convex analysis