Prior Bias in Vision Language Models on UML Diagram Interpretation

2026-07-02

AI Summary

This study investigates whether vision-language models (VLMs) rely on pretraining priors rather than actual diagrammatic relationships when interpreting UML class diagrams. To this end, we construct a controlled diagnostic benchmark comprising paired diagrams that differ only in the direction of relational arrows—specifically contrasting “prior-consistent” and “prior-violating” configurations—to systematically evaluate the relational direction recognition capabilities of prominent VLMs. Experimental results reveal that open-source models such as InternVL3.5 and Qwen3 suffer an average accuracy drop of 33.48% upon arrow reversal, with performance degrading by up to 45.28% across three representative scenarios; GPT-5.4 variants also exhibit significant degradation. This work is the first to uncover and quantitatively characterize prior-induced bias in VLMs’ comprehension of software engineering diagrams and introduces a reproducible diagnostic evaluation methodology.

Abstract

Vision Language Models (VLMs) are increasingly applied to software engineering artifacts, especially UML class diagrams whose meaning depends on visual notation. Yet, it is unclear whether VLMs actually read such diagrams or instead answer from pretrained priors about how classes typically relate. We introduce a controlled UML benchmark in which each prior-conforming diagram is paired with its prior-conflicting counterpart that (1) preserves the same class names and layout while (2) reverses only the relation arrow. We evaluate eight open-source VLMs from two model families, InternVL3.5 and Qwen3, alongside two closed-source frontier models GPT-5.4 and GPT-5.4 Mini. Across the eight open-source models, reversing the arrow reduces relation-direction accuracy by 33.48% on average, while GPT-5.4 Mini retains a 10% gap. In the harder three-class condition, accuracy drops sharply by 45.28% for open-source models, and even 18.62% for the GPT-5.4 family on average. Scaling provides only limited improvements and is family-dependent. Our benchmark presents a diagnostic prior-driven failure in diagram-grounded software understanding. Our artifact is available at https://anonymous.4open.science/r/UMLKnowledgeConflict-8461.

Problem

• Vision Language Models
• UML Diagram Interpretation
• Prior Bias
• Software Engineering Artifacts
• Diagram Understanding

Innovation

• Vision Language Models
• UML Diagram Interpretation
• Prior Bias
• Controlled Benchmark
• Diagram Grounding