Can Text-to-Image Models Draw from the Right Frame of Reference?

2026-08-04

AI Summary

This study addresses the persistent challenge in text-to-image (T2I) generation wherein models frequently conflate egocentric, camera-centric spatial references with object-centric frames of reference, leading to erroneous layouts. To systematically evaluate this issue, the authors introduce FoR-T2I, a benchmark comprising 1,200 carefully curated prompt pairs, and assess 22 state-of-the-art T2I models. Their analysis reveals—for the first time—that model accuracy under object-centric frames of reference is on average 41.8% lower than under camera-centric perspectives, with the best-performing model achieving only 44.3% accuracy. To mitigate this gap, they propose a training-free, VLM-gated prompt rewriting strategy that leverages feedback from a vision-language model to automatically refine spatial instructions, improving average frame-of-reference accuracy from 25.0% to 29.2%.

Abstract

Spatial instruction following has become a crucial requirement for text-to-image (T2I) generation. A common challenge arises when directional expressions are interpreted under different frames of reference. For example, ``the left of'' may refer to the viewer's image coordinates or to the intrinsic orientation of an object, leading to different expected layouts. Existing T2I benchmarks reveal important layout failures, yet they rarely isolate whether models can follow a specified frame of reference when it differs from camera view. To mitigate this gap, we introduce FoR-T2I, a benchmark for evaluating this distinction with 1,200 prompt pairs built from controlled spatial layouts. In each pair, the camera-view (Cam) prompt states the target relation in camera view, while the frame-of-reference (FoR) prompt describes the same target placement through an oriented anchor object. Across 22 closed-source and open-source T2I models, mean final accuracy is 41.8\% lower on FoR prompts than on matched Cam prompts; even the best-performing model achieves only 44.3\% FoR accuracy. This suggests that current models struggle more when the same layout is described through an object's orientation rather than directly in image coordinates. We further analyze this gap by relation type and camera view, compare several training-free prompting and feedback-based mitigation strategies, and propose a VLM-gated rewriting approach that selects rewritten prompts using visual feedback, improving average FoR accuracy from 25.0\% to 29.2\% under the same generation budget.

Problem

• text-to-image generation
• spatial reasoning
• frame of reference
• directional expressions
• layout accuracy

Innovation

• frame of reference
• text-to-image generation
• spatial reasoning
• prompt rewriting
• visual language model