TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

2026-06-02

AI Summary

Current vision-language models exhibit suboptimal performance on education-oriented visual programming tasks, particularly struggling with geometric pattern perception, spatial relationship reasoning, and consistency between visual understanding and code generation. To address this gap, this work introduces TurtleAI, a benchmark comprising 823 real-world tasks, which constitutes the first systematic evaluation of multimodal models’ visual programming capabilities in the Turtle Graphics domain. The study further proposes a synthetic data generation method based on a small set of seed examples to enable few-shot fine-tuning. Experimental results reveal that prevailing models achieve success rates below 30% on TurtleAI; however, after fine-tuning with the proposed strategy, Qwen2-VL-72B demonstrates an approximately 20% performance gain on real tasks, significantly improving alignment between visual reasoning and code implementation.

Abstract

Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.

Problem

• visual programming
• vision-language models
• Turtle Graphics
• spatial reasoning
• code generation

Innovation

• visual programming
• vision-language models
• Turtle Graphics
• synthetic data generation
• spatial reasoning