🤖 AI Summary
Existing benchmarks lack systematic evaluation of HTML element completeness—including invisible elements—and layout structure fidelity in image-to-HTML generation. To address this, we introduce IW-Bench, the first robust benchmark for this task, comprising 1,200 diverse, multi-difficulty image–HTML pairs. We propose two novel, fine-grained metrics: Element Accuracy, computed via DOM parsing to assess semantic completeness, and Layout Accuracy, based on the longest common subsequence over DOM trees to quantify structural fidelity. Further, we design a five-step multimodal chain-of-thought prompting framework integrating Segmentation-of-Multimodality (SoM) injection, element inference, layout modeling, HTML code generation, and reflective optimization. Comprehensive evaluation across state-of-the-art large vision-language models reveals critical deficiencies in both element completeness and relational layout modeling. IW-Bench thus establishes a reproducible, granular evaluation standard and provides actionable insights for advancing image-to-HTML synthesis.
📝 Abstract
Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.