PixVL: Self-Supervised Training of Pixel-Level MLLMs via a Unified Mask--Text Consistency Cycle

2026-08-02

AI Summary

This work addresses the scarcity of high-quality mask–text pairs and the optimization conflict between region segmentation and understanding tasks in pixel-level multimodal large language models, which arises from disparities in supervision formats and signal density. To overcome these challenges, the authors propose PixVL, a self-supervised post-training framework that enables the model to generate and self-verify region descriptions using unlabeled data through a mask–text consistency loop. Key innovations include confusion-aware semantic validation, cross-view consistency checks (e.g., across video frames or geometric transformations), and a quality-coupled bidirectional learning strategy that formulates segmentation and understanding as a synergistic generator–verifier mechanism. Experiments demonstrate that PixVL significantly improves performance on both tasks, effectively mitigating optimization interference while efficiently leveraging unlabeled data.

Abstract

Recent studies develop pixel-level multimodal large language models (MLLMs) that support both Region Segmentation and Region Understanding, extending multimodal interaction from whole images to specific objects and regions. However, these methods face two fundamental challenges. First, the scarcity of high-quality mask--text pairs leaves abundant mask annotations without corresponding language supervision. Second, discrepancies in supervision formats and learning-signal densities induce optimization interference between Region Segmentation and Region Understanding. To address these challenges, we propose PixVL, a self-supervised post-training framework that introduces a unified Mask--Text Consistency Cycle, enabling pixel-level MLLMs to generate and self-verify regional descriptions and learn from unlabeled data. We found that direct cycle based solely on geometric reconstruction is unreliable because re-segmentation IoU does not faithfully reflect the semantic quality and referring sufficiency. PixVL therefore introduces confuser-aware semantic verification, which uses the model's confidence when it correctly chooses the target among highly similar candidate regions, and assigns zero reward to an incorrect choice. Meanwhile, PixVL performs cross-view verification using temporally separated video frames or geometrically transformed image views, preventing cyclic learning from collapsing to positional and shape shortcuts. Finally, a quality-coupled bidirectional learning strategy uses the highest-reward description to guide Text-to-Mask learning. This strategy transforms Region Understanding and Region Segmentation from competing tasks into mutual generators and verifiers. Experiments demonstrate that PixVL improves both region understanding task and segmentation task.

Problem

• pixel-level MLLMs
• mask-text pairs
• Region Segmentation
• Region Understanding
• optimization interference

Innovation

• self-supervised learning
• pixel-level MLLMs
• mask-text consistency
• semantic verification
• bidirectional learning