PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

2026-09-28

AI Summary

This study investigates whether multimodal large language models can exhibit graphical divergent thinking and incremental visual planning within tool-calling scenarios. To this end, the authors adapt incomplete drawing tasks into an agent-based environment, establishing the first agent-oriented benchmark for graphical divergent thinking that requires models to generate original artworks through multi-turn tool invocations. Furthermore, they propose ViDrA-adapted, an automated scorer designed to predict human creativity ratings, and evaluate fourteen mainstream models using this framework. Experimental results demonstrate that GPT-6 Astra achieves the best overall performance. Notably, the findings reveal that while agent-generated artworks exhibit higher creativity than human drawings, they possess lower recognizability. The associated datasets and evaluation frameworks have been made publicly available to facilitate future research.

Abstract

Figural divergent thinking is the ability to develop a given shape fragment into an original drawing. In humans, this ability is assessed with incomplete-drawing tasks. We introduce PainterBench, a benchmark that ports the incomplete-drawing task to the agentic setting. The agent draws on a canvas through tool calls and observes the result after every turn. The canvas includes a starting shape which cannot be erased, and the agent's goal is to incorporate this shape into the most original drawing it can produce. The task is open-ended, and the agent itself decides when the drawing is finished. The benchmark tests incremental visual planning over a short horizon and the transfer of creative ability from pretraining to multi-turn tool use. We evaluate 14 multimodal language models from small to frontier scale. Across the primary study and six sensitivity analyses, we collect 2,700 drawings and crowdsource creativity and recognizability ratings for every drawing and for 300 human reference drawings. We also present ViDrA-adapted, an automated scorer that predicts human creativity ratings of agent drawings (r = 0.85 on random held-out test split). Figural divergent thinking varies widely across the 14 models, and GPT-6 Astra produces the most creative drawings. Relative to the human drawings, the agent drawings score higher in creativity but lower in recognizability. We release the final drawings, per-round canvas snapshots, tool call traces, stimulus bank, benchmark harness, crowdsourced ratings (N = 72,000), and ViDrA checkpoint.

Problem

• figural divergent thinking
• tool-using language models
• multimodal benchmark
• creativity evaluation
• incomplete-drawing task

Innovation

• Figural Divergent Thinking
• Agentic Benchmark
• Tool-Using Language Models
• Automated Creativity Scoring
• Multimodal Evaluation