Problem
Research questions and friction points this paper is trying to address.
Multimodal Large Language Models
On-Policy Self-Distillation
Fine-grained Perception
Synthetic-to-Real Transfer
Spatial Guidance
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
🤖 AI Summary
This study addresses the lack of efficient annotation-free training methods for multimodal large language models in fine-grained perception tasks. To this end, it proposes a spatially guided online self-distillation framework based on synthetic scenes. This framework is the first to leverage programmatically generated scenes to provide annotation-free privileged spatial information, integrating textual spatial guidance with coordinate supervision mechanisms to perform online policy self-distillation, thereby achieving significant transfer from synthetic data to real-world scenarios. Experimental results demonstrate that the proposed method yields an average performance improvement of 3.23 points across multiple benchmarks, substantially enhancing the model's counting and document understanding capabilities.
📝 Abstract
On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a frozen or EMA version of themselves that receives privileged information. Its application to multimodal large language models (MLLMs), however, remains largely unexplored. Recent approaches use privileged visual information, such as image crops corresponding to a question, to improve fine-grained perception, but their gains are confined to tasks that benefit from such visual zooming and require either human-annotated grounding data or external teacher models. We introduce a different form of on-policy self-distillation for MLLMs that provides the teacher with textual, spatially grounded guidance identifying the visual elements relevant to a query. We use procedurally generated scenes with automatically available object identities and spatial coordinates, enabling scalable and annotation-free post-training. The teacher uses this spatial guidance to locate and integrate evidence from multiple relevant image regions, while the student learns to reproduce the resulting behavior from the image and question alone. Our approach consistently improves performance on counting, document and chart understanding benchmarks across multiple models. Importantly, although post-training uses only synthetic scenes, the resulting improvements transfer to real-world perception benchmarks, yielding a 3.23-point gain in average performance across CVBench, V*, ZoomBench, BLINK, HR-Bench, and MME-RealWorld. These results show that spatially grounded privileged information can induce broader perceptual capabilities through on-policy self-distillation, enabling substantial synthetic-to-real transfer beyond the task and data distribution used for post-training. Project page: https://github.com/sirkosophia/Where-OPD
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-policy self-distillation
Multimodal large language models
Spatial guidance
Synthetic-to-real transfer
Procedural generation
🔎 Similar Papers
No similar papers found.
