Explaining Image Similarity with Automatically Extracted Concept Activation Vectors

2026-07-30

AI Summary

This work addresses the lack of interpretability in existing image similarity scoring methods with respect to global semantic factors such as texture, shape, and color. The authors propose a model- and metric-agnostic framework that leverages sparse autoencoders to automatically extract Concept Activation Vectors (CAVs). By perturbing image representations along concept directions in the embedding space, the method quantifies each semantic concept’s contribution to similarity and generates concept attribution maps for spatial localization. The approach supports similarity explanations ranging from individual image pairs to group-level clustering and introduces a similarity-based example retrieval mechanism grounded in causal attribution. Experiments demonstrate that latent-space perturbations better conform to the underlying data distribution, that concept importance scores can linearly reconstruct ground-truth similarity, and that both qualitative and quantitative results validate the framework’s effectiveness at individual and population levels.

Abstract

Image similarity underlies many computer vision applications, yet it is often unclear why two images receive a high or low similarity score. Existing explainability methods often rely on gradient-based attribution maps to provide local justifications for similarity. These approaches struggle to provide global insights into what specifically drives similarity in regions of an embedding space, such as texture, shape, or color. We introduce a model- and metric-agnostic framework that explains image similarity using Concept Activation Vectors (CAVs) extracted automatically via Sparse Autoencoders (SAEs). Given a pair of images, we perturb their embeddings along discovered concept directions and measure the resulting change in a chosen similarity function, yielding concept importances. For image pairs, we provide localization with concept attribution maps. We extend this procedure to group-level settings, explaining what drives similarity across a cluster of images rather than a single pair, and further, we introduce Exemplar Retrieval, aiming to recover samples with similar reasons contributing to similarity. Our experiments show that our latent perturbations are more faithful to the underlying data distribution than pixel-space baselines, and that concept importances linearly recover the true similarity score. Qualitative results further confirm the usefulness of our methods in understanding a model's individual and group similarity judgments.

Problem

• image similarity
• explainability
• concept activation
• embedding space
• global interpretation

Innovation

• Concept Activation Vectors
• Sparse Autoencoders
• Similarity Explanation
• Model-agnostic Interpretability
• Exemplar Retrieval