MLLM-Based UI2Code Automation Guided by UI Layout Information

2025-06-12

AI Summary

UI-to-Code (UI2Code) faces challenges including heavy reliance on large-scale annotated data, poor generalization, and difficulty for multimodal large language models (MLLMs) to accurately model complex UI layout structures. Method: This paper proposes LayoutCoder—a novel framework that explicitly incorporates layout structure awareness into the MLLM-driven generation pipeline. It comprises three core modules: visual element relation construction, UI layout tree parsing, and layout-guided code fusion. The approach integrates visual component detection, hierarchical layout tree generation, layout constraint injection, and code sequence fusion. Contribution/Results: Evaluated on our newly constructed Snap2Code benchmark (350 real-world websites) and the Design2Code benchmark, LayoutCoder significantly outperforms state-of-the-art methods—achieving +10.14 BLEU and +3.95 CLIP score improvements. It enables end-to-end, layout-preserving, highly generalizable, and low-data-dependent generation of frontend code from webpage screenshots.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Converting user interfaces into code (UI2Code) is a crucial step in website development, which is time-consuming and labor-intensive. The automation of UI2Code is essential to streamline this task, beneficial for improving the development efficiency. There exist deep learning-based methods for the task; however, they heavily rely on a large amount of labeled training data and struggle with generalizing to real-world, unseen web page designs. The advent of Multimodal Large Language Models (MLLMs) presents potential for alleviating the issue, but they are difficult to comprehend the complex layouts in UIs and generate the accurate code with layout preserved. To address these issues, we propose LayoutCoder, a novel MLLM-based framework generating UI code from real-world webpage images, which includes three key modules: (1) Element Relation Construction, which aims at capturing UI layout by identifying and grouping components with similar structures; (2) UI Layout Parsing, which aims at generating UI layout trees for guiding the subsequent code generation process; and (3) Layout-Guided Code Fusion, which aims at producing the accurate code with layout preserved. For evaluation, we build a new benchmark dataset which involves 350 real-world websites named Snap2Code, divided into seen and unseen parts for mitigating the data leakage issue, besides the popular dataset Design2Code. Extensive evaluation shows the superior performance of LayoutCoder over the state-of-the-art approaches. Compared with the best-performing baseline, LayoutCoder improves 10.14% in the BLEU score and 3.95% in the CLIP score on average across all datasets.

Problem

• Automating UI-to-code conversion to improve development efficiency
• Overcoming generalization issues with unseen web page designs
• Enhancing MLLMs' ability to preserve complex UI layouts in code

Innovation

• MLLM-based UI2Code automation with layout guidance
• Element Relation Construction for UI component grouping
• Layout-Guided Code Fusion preserving UI structure