Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning

2026-07-16

AI Summary

Existing multimodal large language models struggle to accurately localize object parts under free-form linguistic queries due to their lack of explicit modeling of object–part hierarchical structure. This work proposes Object–Part Hierarchical Reflective Grounding (OP-HRG), a coarse-to-fine reasoning approach that first localizes the parent object and then precisely identifies its constituent parts within the detected region. To enhance localization accuracy, OP-HRG incorporates a self-reflective correction mechanism and region re-encoding. Furthermore, we introduce a GRPO-based reinforcement learning framework with stage-wise rewards, enabling, for the first time, joint optimization of hierarchical object–part grounding and self-reflection. The proposed 4B-parameter model outperforms 7B-scale grounding LLMs and SAM3 on PascalPart, PartImageNet, and InstructPart, and demonstrates successful transfer to referring segmentation tasks.

Abstract

Multimodal large language models (MLLMs) ground whole objects well from free-form language queries, but they struggle when the query names a part rather than the object. We trace this to a missing object-part hierarchy, since parts are localized in the same single step used for objects. We propose Object-Part Hierarchical Reflective Grounding (OP-HRG), a coarse-to-fine reasoning-guided grounding strategy that first localizes the parent object and then the part within it. A self-check then reflects on the result, with an extension to re-encode the predicted crop to inspect the region it is correcting. We introduce a part-aware GRPO framework to train our pipeline with stage-wise rewards. A 4B model trained this way outperforms 7B grounding LLMs and SAM3 across PascalPart, PartImageNet, and InstructPart, and transfers to reasoning segmentation.

Problem

• visual grounding
• object-part hierarchy
• part-level grounding
• multimodal language models
• reasoning

Innovation

• part-level visual grounding
• hierarchical reasoning
• reinforcement learning
• self-reflection mechanism
• GRPO