Concept Retrieval - What and How?

2025-10-08

AI Summary

This paper introduces the novel task of *concept retrieval*: given a query image, retrieve other images that share a deep, narrative-level conceptual core—not merely visual or semantic similarity. Methodologically, we propose a bimodal Gaussian distribution model to characterize neighborhood structure in embedding space; leverage concept consistency metrics to identify latent, mutually exclusive concepts shared across distinct neighborhoods; and integrate automated evaluation with human annotation for multidimensional validation. Experiments across multiple benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art image retrieval and clustering methods in qualitative analysis, quantitative metrics, and human evaluation. To our knowledge, this is the first work to formally define the concept retrieval task, establish its theoretical foundations, and provide a reproducible, multi-faceted evaluation framework.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

A concept may reflect either a concrete or abstract idea. Given an input image, this paper seeks to retrieve other images that share its central concepts, capturing aspects of the underlying narrative. This goes beyond conventional retrieval or clustering methods, which emphasize visual or semantic similarity. We formally define the problem, outline key requirements, and introduce appropriate evaluation metrics. We propose a novel approach grounded in two key observations: (1) While each neighbor in the embedding space typically shares at least one concept with the query, not all neighbors necessarily share the same concept with one another. (2) Modeling this neighborhood with a bimodal Gaussian distribution uncovers meaningful structure that facilitates concept identification. Qualitative, quantitative, and human evaluations confirm the effectiveness of our approach. See the package on PyPI: https://pypi.org/project/coret/

Problem

• Retrieving images sharing central concepts beyond visual similarity
• Modeling neighborhood structure with bimodal Gaussian distributions
• Developing evaluation metrics for concept-based image retrieval

Innovation

• Retrieves images sharing central concepts beyond visual similarity
• Models neighborhood with bimodal Gaussian distribution for structure
• Uses embedding space neighbors to identify shared concepts