TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

2026-05-31

AI Summary

Existing reinforcement learning approaches for visual reasoning are constrained by static datasets, lacking scalable, controllable, and verifiable training signals. This work proposes TRON—an online environment framework that leverages a procedural generator-verifier architecture to dynamically produce training samples comprising novel visual states, images, questions, and exact answers. TRON encompasses 520 environments across five categories: spatial reasoning, mathematics, diagrams, pattern logic, and counting. It enables, for the first time, the generation of unlimited, rule-verifiable visual reasoning instances, supporting dynamic curriculum learning and unified training of both general-purpose and specialized models without additional data collection. Experiments demonstrate significant performance gains on Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT, consistently outperforming ten external multimodal reasoning benchmarks.

Abstract

Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.

Problem

• visual reasoning
• reinforcement learning
• training signals
• static datasets
• scalability

Innovation

• online environment
• rule-verifiable generation
• visual reasoning RL
• curriculum-controlled training
• dynamic instance generation