Visualization Biases MLLM's Decision Making in Network Data Tasks

2025-11-05

AI Summary

This study investigates how visualizations influence multimodal large language models’ (MLLMs) reasoning about the existence of “bridge edges” in network graphs. We compare MLLM performance under two input modalities: structured textual descriptions versus standard graph visualizations (e.g., force-directed layouts), while holding underlying graph topology constant. Contrary to expectations, although visualizations substantially increase model confidence, they induce strong systematic biases—MLLMs develop topology-agnostic preferences for either affirming or denying bridge existence, resulting in degraded accuracy. This reveals a novel class of visualization-induced hallucinations: user-provided diagrams can mislead MLLMs into producing highly confident yet factually incorrect judgments. The work provides the first empirical identification and quantification of implicit bias mechanisms introduced by graph visualizations in MLLM reasoning. It offers critical insights and methodological foundations for designing safe, reliable visualization–language interfaces in generative AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Preferences

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data visualization
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We evaluate how visualizations can influence the judgment of MLLMs about the presence or absence of bridges in a network. We show that the inclusion of visualization improves confidence over a structured text-based input that could theoretically be helpful for answering the question. On the other hand, we observe that standard visualization techniques create a strong bias towards accepting or refuting the presence of a bridge -- independently of whether or not a bridge actually exists in the network. While our results indicate that the inclusion of visualization techniques can effectively influence the MLLM's judgment without compromising its self-reported confidence, they also imply that practitioners must be careful of allowing users to include visualizations in generative AI applications so as to avoid undesired hallucinations.

Problem

• Visualizations bias MLLM decisions in network bridge identification tasks
• Standard visualization techniques create false bridge presence judgments
• Visualizations influence MLLM confidence without improving factual accuracy

Innovation

• Visualization biases MLLM decisions in network analysis
• Standard techniques create bias regardless of actual bridge presence
• Visualizations influence judgment without reducing model confidence