The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue

2026-06-01

AI Summary

This work proposes a multi-turn multimodal dialogue framework that iteratively reconstructs images through interactive exchanges between a vision-language model (the describer) and an image generator, explicitly embedding the shared understanding developed during dialogue into the final output. It pioneers the integration of multi-round visual-language dialogue with image generation by establishing a fully automated benchmark task based on an instruction-generation-correction loop, complemented by automated evaluation metrics calibrated against human preferences. The study reveals that mathematical and geometric images are the most challenging to reconstruct; the describer plays a dominant role in reconstruction quality, with its token budget influencing convergence behavior, and stronger describers more effectively leveraging spatial, numerical, and structural vocabulary. Furthermore, current automatic scorers exhibit only weak-to-moderate alignment with human preferences, underscoring the necessity of human-in-the-loop calibration.

Abstract

We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image. Benchmarking two Describer models crossed with two Generator models across seven image categories, we find that the describer is the dominant factor in reconstruction quality, while the generator determines whether iterative refinement helps or hurts. Mathematical and geometric images pose the greatest challenge. The describer's token budget strongly affects convergence: shorter budgets yield sparser first renderings with more room for visible improvement, while longer budgets raise absolute quality but leave less to fix. Stronger describers use a richer correction vocabulary spanning spatial, numeric, and structural categories, while weaker describers concentrate on surface properties and tend to stop after a few turns. Human validation shows that the best automated judge reaches only slight-to-fair agreement with human preferences, and automated scores require human recalibration to be used reliably.

Problem

• image reconstruction
• multimodal dialogue
• common ground
• vision-language model
• iterative refinement

Innovation

• Image Reconstruction Game
• multimodal dialogue
• common ground
• iterative refinement
• vision-language model