Can VLMs Assess Similarity Between Graph Visualizations?

2025-04-14

AI Summary

This work investigates whether vision-language models (VLMs) can reliably replace human annotation for assessing visual similarity among graph visualizations and examines their consistency with traditional feature-based graph similarity measures. To this end, the authors construct a multi-scale density plot dataset and compute visual similarity via image embeddings extracted from VLMs—including CLIP and FLAVA—followed by cosine similarity computation. These results are systematically compared against six established feature-based metrics, such as Graph Edit Distance and Spectral Distance. The study provides the first systematic empirical validation showing that VLM-derived visual similarity exhibits significant moderate-to-strong correlations (Spearman ρ = 0.62–0.79) with multiple feature-based methods, without relying on hand-crafted features. This demonstrates that VLMs offer a scalable, annotation-free paradigm for evaluating graph visualizations and extends the foundational applicability of VLMs to graph data analysis.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Graph visualizations have been studied for tasks such as clustering and temporal analysis, but how these visual similarities relate to established graph similarity measures remains unclear. In this paper, we explore the potential of Vision Language Models (VLMs) to approximate human-like perception of graph similarity. We generate graph datasets of various sizes and densities and compare VLM-derived visual similarity scores with feature-based measures. Our findings indicate VLMs can assess graph similarity in a manner similar to feature-based measures, even though differences among the measures exist. In future work, we plan to extend our research by conducting experiments on human visual graph perception.

Problem

• Explore VLMs' ability to assess graph visualization similarity
• Compare VLM-derived scores with feature-based similarity measures
• Investigate alignment between visual and traditional graph similarity metrics

Innovation

• VLMs approximate human-like graph similarity perception
• Compare VLM scores with feature-based measures
• Generate diverse graph datasets for analysis