IoUPD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

2026-07-17

AI Summary

This work addresses the misalignment between training objectives—typically token-level likelihood of coordinate strings—and evaluation metrics such as Intersection over Union (IoU) in visual grounding. To bridge this gap, the authors propose IoUPD, a novel approach that uniquely leverages ground-truth bounding boxes both as coordinate labels and as privileged prompts during training. IoUPD introduces an IoU-aware weighted distillation loss that guides multimodal large language models to generate more accurate localization coordinates without requiring additional modules at inference time. By integrating autoregressive coordinate prediction, box-augmented visual prompting, and geometry-aware token-level loss weighting, IoUPD achieves consistent and significant improvements over strong baselines on standard referring expression comprehension benchmarks, demonstrating superior region-level localization performance.

Abstract

Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoUPD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoUPD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoUPD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels.

Problem

• visual grounding
• multimodal large language models
• coordinate generation
• training-evaluation mismatch
• geometric overlap

Innovation

• privileged distillation
• IoU-aware
• visual grounding
• multimodal large language models
• coordinate generation