🤖 AI Summary
Current multimodal large language models (MLLMs) are limited to image-level or bounding-box-level understanding, lacking pixel-level vision-language alignment capabilities, primarily due to the scarcity of mask-text instruction data. To address this, we propose a novel mask-text instruction fine-tuning paradigm—the first to enable pixel-level, fine-grained cross-modal joint modeling. Our approach introduces a mask-driven visual instruction fine-tuning framework, constructs a high-quality mask-text instruction dataset comprising 724K samples, and incorporates a mask-aware visual feature extractor with pixel-level representation injection—fully compatible with SAM for multi-granularity semantic understanding. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art MLLMs across multiple region-understanding benchmarks, achieving superior capabilities in precise pixel localization, mask generation, and cross-modal referential grounding.
📝 Abstract
Multimodal large language models (MLLMs) have recently achieved impressive general-purpose vision-language capabilities through visual instruction tuning. However, current MLLMs primarily focus on image-level or box-level understanding, falling short in achieving fine-grained vision-language alignment at pixel level. Besides, the lack of mask-based instruction data limits their ad-vancements. In this paper, we propose Osprey, a mask-text instruction tuning approach, to extend MLLMs by incor-porating fine-grained mask regions into language instruction, aiming at achieving pixel-wise visual understanding. To achieve this goal, we first meticulously curate a mask-based region-text dataset with 724K samples, and then design a vision-language model by injecting pixel-level representation into LLM. Specifically, Osprey adopts a convolutional CLIP backbone as the vision encoder and employs a mask-aware visual extractor to extract precise visual mask features from high resolution input. Experimen-tal results demonstrate Osprey's superiority in various region understanding tasks, showcasing its new capability for pixel-level instruction tuning. In particular, Osprey can be integrated with Segment Anything Model (SAM) seamlessly to obtain multi-granularity semantics. The source code, dataset and demo can be found at https://github.com/CircleRadon/Osprey.
