Technology Category
Machine Learning: ClusteringSearch and Optimization: Heuristic SearchConstraint Satisfaction and Optimization: Distributed CSP/Optimization
Strong bounds for large-scale Minimum Sum-of-Squares Clustering
🤖 AI Summary
Evaluating the quality of heuristic solutions for large-scale Minimum Sum-of-Squares Clustering (MSSC) remains challenging due to the difficulty in obtaining tight lower bounds. Method: This paper proposes an exact lower-bound computation method for MSSC, integrating a divide-and-conquer framework guided by anti-clustering heuristics. Anti-clustering is innovatively employed as a data decomposition mechanism; combined with Euclidean-distance-based optimization modeling and exact mixed-integer programming (MIP) solving, it efficiently generates strong lower bounds within the divide-and-conquer scheme. Contribution/Results: The method achieves, for the first time on large-scale MSSC instances, stable optimality gaps below 3% with controllable computational time. It significantly enhances the feasibility and reliability of optimality verification for clustering solutions, establishing a novel, verifiable quality-assurance paradigm for large-scale unsupervised learning.
Technology Category
Application Category
📝 Abstract
Clustering is a fundamental technique in data analysis and machine learning, used to group similar data points together. Among various clustering methods, the Minimum Sum-of-Squares Clustering (MSSC) is one of the most widely used. MSSC aims to minimize the total squared Euclidean distance between data points and their corresponding cluster centroids. Due to the unsupervised nature of clustering, achieving global optimality is crucial, yet computationally challenging. The complexity of finding the global solution increases exponentially with the number of data points, making exact methods impractical for large-scale datasets. Even obtaining strong lower bounds on the optimal MSSC objective value is computationally prohibitive, making it difficult to assess the quality of heuristic solutions. We address this challenge by introducing a novel method to validate heuristic MSSC solutions through optimality gaps. Our approach employs a divide-and-conquer strategy, decomposing the problem into smaller instances that can be handled by an exact solver. The decomposition is guided by an auxiliary optimization problem, the"anticlustering problem", for which we design an efficient heuristic. Computational experiments demonstrate the effectiveness of the method for large-scale instances, achieving optimality gaps below 3% in most cases while maintaining reasonable computational times. These results highlight the practicality of our approach in assessing feasible clustering solutions for large datasets, bridging a critical gap in MSSC evaluation.
Problem
Research questions and friction points this paper is trying to address.
Improving large-scale MSSC optimality validation
Reducing computational complexity in MSSC
Enhancing heuristic solution quality assessment
Innovation
Methods, ideas, or system contributions that make the work stand out.
Divide-and-conquer strategy
Optimality gaps validation
Auxiliary anticlustering heuristic
🔎 Similar Papers
2024-10-08arXiv.orgCitations: 0
