Learning Self-Growth Maps for Fast and Accurate Imbalanced Streaming Data Clustering

2024-04-14

AI Summary

To address the dual challenges of concept drift and cluster-size imbalance in streaming data clustering—which degrade both accuracy and computational efficiency—this paper proposes the Self-Growing Map (SGM) model and the SOHI hierarchical merging strategy. SGM constructs a density-sensitive, on-demand growing neural topology that robustly identifies small clusters and incrementally adapts to evolving cluster structures. SOHI performs hierarchical cluster merging via local neighborhood retrieval, eliminating costly global search while preserving clustering accuracy and substantially improving runtime efficiency. The method integrates density-driven neuron allocation, SGM-guided incremental learning, and localized retrieval. Evaluated on multiple imbalanced streaming datasets, it achieves an average 12.7% improvement in clustering accuracy, a 3.2× speedup in execution time, and stable estimation of the true number of clusters—effectively mitigating small-cluster omission.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Clustering
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Streaming data clustering is a popular research topic in data mining and machine learning. Since streaming data is usually analyzed in data chunks, it is more susceptible to encounter the dynamic cluster imbalance issue. That is, the imbalance ratio of clusters changes over time, which can easily lead to fluctuations in either the accuracy or the efficiency of streaming data clustering. Therefore, we propose an accurate and efficient streaming data clustering approach to adapt the drifting and imbalanced cluster distributions. We first design a Self-Growth Map (SGM) that can automatically arrange neurons on demand according to local distribution, and thus achieve fast and incremental adaptation to the streaming distributions. Since SGM allocates an excess number of density-sensitive neurons to describe the global distribution, it can avoid missing small clusters among imbalanced distributions. We also propose a fast hierarchical merging strategy to combine the neurons that break up the relatively large clusters. It exploits the maintained SGM to quickly retrieve the intra-cluster distribution pairs for merging, which circumvents the most laborious global searching. It turns out that the proposed SGM can incrementally adapt to the distributions of new chunks, and the Self-grOwth map-guided Hierarchical merging for Imbalanced data clustering (SOHI) approach can quickly explore a true number of imbalanced clusters. Extensive experiments demonstrate that SOHI can efficiently and accurately explore cluster distributions for streaming data.

Problem

• Address dynamic cluster imbalance in streaming data clustering
• Adapt to drifting and imbalanced cluster distributions efficiently
• Accurately identify small clusters in imbalanced data streams

Innovation

• Self-Growth Map adapts dynamically to data chunks
• Density-sensitive neurons prevent missing small clusters
• Hierarchical merging strategy avoids global searching