SG-Layout: Structured Scene Graph-Guided Layout Generation with LLMs

2026-08-02

AI Summary

This work addresses the challenge that large language models face in comprehending spatial relationships in natural language and generating geometrically consistent layouts. The authors propose SG-Layout, a novel framework that explicitly incorporates structured scene graphs into large language models for the first time. By employing a graph encoder and a projector to align graph and language features, and leveraging LoRA for efficient instruction tuning while keeping the backbone network frozen, SG-Layout significantly enhances spatial reasoning accuracy and geometric consistency. The method demonstrates strong performance across diverse tasks—including image layout generation, indoor scene synthesis, and robotic object rearrangement—particularly excelling in scenarios characterized by dense relational structures and complex compositional arrangements.

Abstract

Understanding and generating spatially coherent layouts from natural language remains a fundamental yet challenging task for large language models (LLMs). Existing LLMs often struggle to capture explicit geometric relationships and structural dependencies between objects. To address this issue, we propose SG-Layout, a graph-guided layout generation framework that explicitly incorporates structured spatial knowledge into LLMs. SG-Layout follows a two-stage training paradigm: (1) a graph-language feature alignment stage, where a relational graph encoder and a projector are trained to map scene-graph embeddings into the LLM's linguistic space; and (2) an instruction tuning stage, where LoRA-based adapters enable efficient fine-tuning for instruction-driven layout generation while keeping the backbone frozen. We evaluate SG-Layout on image layout generation, indoor scene synthesis and robotic object rearrangement tasks. Experimental results show that SG-Layout improves spatial reasoning accuracy and geometric consistency over the compact open-source backbone, with particularly clear advantages in relation-dense and compositionally complex scenes. These results highlight the effectiveness of graph-structured feature alignment for enhancing controllable layout generation.

Problem

• layout generation
• spatial reasoning
• scene graph
• geometric relationships
• large language models

Innovation

• scene graph
• layout generation
• large language models
• spatial reasoning
• LoRA