The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

2026-07-20

AI Summary

This work addresses the limitation of existing image perceptual similarity metrics in capturing the context-dependent nature of human judgments across diverse semantic dimensions—such as shape or color. To this end, the authors construct a large-scale dataset of human triplet similarity judgments annotated with free-form semantic dimensions and leverage it to fine-tune state-of-the-art vision-language models. They propose TPIPS (Text-Prompted Perceptual Image Similarity), a novel metric that enables dynamic specification of similarity semantics via natural language prompts, thereby overcoming the rigidity of conventional single-purpose similarity measures. Experiments demonstrate that TPIPS significantly outperforms existing methods in aligning with human judgments and generalizing across data distributions. The approach also proves effective in downstream applications including text-guided image retrieval, compositional search, and fine-grained evaluation of generative models.

Abstract

Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS

Problem

• visual similarity
• perceptual metric
• context-dependent judgment
• semantic aspects
• image triplets

Innovation

• text-prompted similarity
• perceptual metric
• vision-language model
• context-dependent similarity
• fine-grained evaluation