Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

2026-08-03

AI Summary

This work addresses the challenge of balancing accuracy, instruction-following capability, and inference efficiency in multimodal large language models for segmentation tasks. The authors propose STAMPlus, a structured fully masked prediction framework that decouples autoregressive dialogue from non-autoregressive mask prediction, enabling parallel generation of multiple explicitly ID-tagged target masks within a single forward pass. Key innovations include a <SEG> trigger mechanism, image-aligned mask token fusion, hybrid attention-based classification, and shared multi-class mask space binding, collectively supporting diverse segmentation scenarios—such as referring expression, open-vocabulary semantics, instance-aware parsing, and small-object remote sensing—while overcoming single-target limitations. Experiments demonstrate state-of-the-art performance across multiple benchmarks, preserved general-purpose multimodal instruction-following ability, and a significant reduction in inference latency for 12-class segmentation from 13.50 seconds to 5.16 seconds.

Abstract

MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.

Problem

• MLLM-based segmentation
• segmentation trilemma
• non-autoregressive mask prediction
• multi-target segmentation
• multimodal instruction following

Innovation

• All-Mask Prediction
• non-autoregressive segmentation
• structured mask representation
• multimodal large language models
• efficient inference