Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

2026-08-04

AI Summary

This work addresses the susceptibility of multimodal large language models (MLLMs) to repetitive UI patterns in the screenshot-to-code task, revealing their tendency to generate visually incorrect yet pattern-consistent code. The study introduces the first evaluation benchmark specifically designed to assess visual pattern completion bias, formulating a fill-in-the-blank screenshot-to-code task. Leveraging the Design2Code dataset, the authors construct 1,440 perturbed samples covering variations in card layouts and text styles to systematically evaluate models’ ability to recover ground-truth attributes when local elements are altered. Experiments demonstrate that five state-of-the-art MLLMs exhibit substantial bias, with bias rates ranging from 69.78% to 80.22% and accuracy as low as 7.89% to 21.17%, highlighting their overreliance on recurring visual patterns—even when anomalies are detected, models prefer outputs that conform to learned structural regularities.

Abstract

Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. In this work, we test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from the Design2Code dataset, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. Codex-5.3 performs best but still drops from 68.61% accuracy on cards to 13.89% on text, while Flash-3.0 reaches 96.11% bias on text. Noise, subtler perturbations, and boundary positions further increase bias rate. Reasoning analysis further shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer. Our results identify a concrete failure mode in multimodal code generation and show that its severity is strongly associated with visual saliency

Problem

• multimodal code generation
• pattern completion bias
• UI patterns
• screenshot-to-code
• visual saliency

Innovation

• pattern-completion bias
• multimodal code generation
• screenshot-to-code
• visual saliency
• UI pattern perturbation