Reinforcement Learning from Intermediate Renders for Image-to-Code Generation

2026-09-28

AI Summary

This study addresses the challenges of sparse terminal rewards and difficult token-level contribution alignment in image-to-code generation by proposing the IR4RL framework. This method leverages intermediate rendering results to provide dense supervision, introducing a novel token-level reward mechanism based on intermediate rendering progress that generates localized feedback through comparisons of intermediate state changes for process supervision. By integrating reinforcement learning post-training with an improved Group Relative Policy Optimization (GRPO) algorithm, the framework effectively optimizes vision-language models. Experimental results demonstrate that this approach significantly outperforms supervised fine-tuning (SFT) and standard GRPO baselines on SVG and TikZ generation tasks, establishing new state-of-the-art performance records among open-source models.

Abstract

Reinforcement learning is increasingly used to post-train vision-language models for image-to-code generation, such as generating SVG code from a reference image, by optimizing rewards computed from the final rendered output. However, relying on a single terminal reward provides sparse feedback that is poorly aligned with the contribution of individual tokens. A generated program may contain operations that accurately reproduce some parts of the target image alongside others that introduce errors, yet all tokens are trained from the same final outcome. We observe that many intermediate code prefixes are not only executable, but already produce meaningful partial renders that reflect progress toward the target. This property provides a natural source of denser supervision during generation. Based on this observation, we introduce IR4RL, an RL framework with a token-level render-progress reward that turns changes between intermediate renders into localized feedback for the generated sequence. We evaluate our approach on Image-to-SVG and Image-to-TikZ generation. Across both tasks, our method improves over supervised fine-tuning and standard GRPO, yielding new state-of-the-art open-source models. This shows that intermediate rendering provides a simple and effective source of process supervision for RL post-training of image-to-code models.

Problem

• Image-to-Code Generation
• Reinforcement Learning
• Sparse Reward
• Token-level Feedback
• Vision-Language Models

Innovation

• Reinforcement Learning
• Intermediate Renders
• Image-to-Code Generation
• Process Supervision
• Token-level Reward