CluStRE: Streaming Graph Clustering with Multi-Stage Refinement

2025-02-08

AI Summary

To address the challenge of efficient, high-quality community detection in large-scale dynamic graph streams, this paper proposes a multi-stage refinement streaming graph clustering method. The core innovation lies in (1) constructing and dynamically evolving a quotient graph within the streaming setting for the first time, (2) integrating modularity optimization with a re-streaming mechanism, and (3) introducing an evolution-inspired heuristic alongside a multi-configuration adaptive scheduling strategy. Our method achieves clustering quality exceeding 96% of the offline Louvain algorithm—despite requiring over one-third less memory. Compared to state-of-the-art streaming methods, it improves clustering quality by up to 89.8% (and up to 150% under optimal configuration) while accelerating execution by 2.6×. These advances substantially narrow the performance gap between streaming and batch-mode graph clustering.

Technology Category

Machine Learning: ClusteringData Mining & Knowledge Management: Data Stream MiningSearch and Optimization: Distributed Search

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsWeb Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web textEconomics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present CluStRE, a novel streaming graph clustering algorithm that balances computational efficiency with high-quality clustering using multi-stage refinement. Unlike traditional in-memory clustering approaches, CluStRE processes graphs in a streaming setting, significantly reducing memory overhead while leveraging re-streaming and evolutionary heuristics to improve solution quality. Our method dynamically constructs a quotient graph, enabling modularity-based optimization while efficiently handling large-scale graphs. We introduce multiple configurations of CluStRE to provide trade-offs between speed, memory consumption, and clustering quality. Experimental evaluations demonstrate that CluStRE improves solution quality by 89.8%, operates 2.6 times faster, and uses less than two-thirds of the memory required by the state-of-the-art streaming clustering algorithm on average. Moreover, our strongest mode enhances solution quality by up to 150% on average. With this, CluStRE achieves comparable solution quality to in-memory algorithms, i.e. over 96% of the quality of clustering approaches, including Louvain, effectively bridging the gap between streaming and traditional clustering methods.

Problem

• Streaming graph clustering algorithm
• Balances efficiency with clustering quality
• Reduces memory overhead significantly

Innovation

• Streaming graph clustering algorithm
• Multi-stage refinement technique
• Quotient graph for modularity optimization