Hierarchical Semantic Alignment for Image Clustering

2025-11-30

AI Summary

Existing image clustering methods that rely on nouns as external semantic knowledge suffer from noun ambiguity, leading to distorted semantic representations and degraded clustering performance. To address this, we propose a training-free hierarchical semantic alignment framework. First, we select semantically relevant nouns via WordNet and integrate fine-grained image-text descriptions with high-level noun concepts to construct multi-granularity textual semantics. Then, we employ optimal transport to align image features with these multi-level textual semantics layer-by-layer. To our knowledge, this is the first work to jointly introduce multi-granularity textual semantic modeling and optimal transport into unsupervised image clustering, effectively mitigating noun ambiguity. Extensive experiments on eight benchmark datasets demonstrate its effectiveness: on ImageNet-1K, it achieves +4.2% accuracy and +2.9% adjusted Rand index over state-of-the-art training-free methods.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Clustering
• Computer Vision: Representation Learning for Vision

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clus- tering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves cluster- ing performance in a training-free manner. In our approach, we incorporate two complementary types of textual seman- tics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.

Problem

• Addresses noun ambiguity in image clustering semantics
• Aligns image features with textual nouns and captions
• Improves clustering accuracy without training requirements

Innovation

• Hierarchical semantic alignment using nouns and captions
• Optimal transport for aligning image features with text
• Training-free method combining enhanced semantic and image features