Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?

2026-07-31

AI Summary

This work addresses the challenge of evaluating multimodal large language models (MLLMs) in zero-shot salient object detection, where existing methods rely on task-specific supervision and fail to disentangle localization from segmentation capabilities. The authors decouple the task into two stages—localization and segmentation—and introduce SaliLLM, a novel benchmark comprising phrases, bounding boxes, and attributes. They propose FOCUS, a training-free framework that integrates top-down Bayesian surprise calibration with bottom-up MLLM evidence propagation, guided by a Gestalt-inspired cooperative attention mechanism to achieve consensus-aligned foreground organization. Their analysis reveals, for the first time, a capability mismatch in MLLMs: strong at localization yet weak at segmentation. FOCUS significantly outperforms state-of-the-art methods across 13 RGB, RGB-D, and RGB-T benchmarks, reducing mean absolute error by 11%, 34%, and 48% compared to fully supervised, weakly supervised, and self-supervised approaches, respectively.

Abstract

The zero-shot capabilities of multimodal large language models (MLLMs) are pushing salient object detection (SOD) beyond task-specific supervision. To disentangle MLLMs beyond conventional mask-based evaluation, we decompose SOD into localization and segmentation, and re-engineer datasets with phrases, boxes, and attributes, establishing a diagnostic benchmark for MLLM saliency perception (SaliLLM). SaliLLM uncovers a striking capability mismatch: MLLMs outperform state-of-the-art (SOTA) methods in localization, yet remain substantially weaker in segmentation. Further analyses attribute this gap primarily to mismatches between MLLMs and annotations over foreground cardinality, granularity, and extent. Motivated by this diagnosis, we recast zero-shot SOD as protocol-aligned Foreground Organization and introduce the first training-free framework that leverages Gestalt-inspired Collaborative attention for Unified SOD (FOCUS). FOCUS couples top-down Bayesian-surprise calibration of protocol-conditioned foreground granularity with bottom-up propagation of MLLMs evidence over entity-centric perceptual manifolds induced by self-supervised features, yielding coherent object extents as prompts for a general segmenter. Across 13 RGB, RGB-D, and RGB-T SOD benchmarks, FOCUS generally surpasses SOTA methods without training, reducing mean absolute error by 11\%, 34\%, and 48\% compared with fully, weakly, and self-supervised methods, respectively. Our findings signal the renaissance of SOD: from task-specific supervision to zero-shot foreground organization. Code is available in the supplementary material.

Problem

• salient object detection
• multimodal large language models
• zero-shot learning
• segmentation
• localization

Innovation

• zero-shot salient object detection
• multimodal large language models
• foreground organization
• training-free framework
• Gestalt-inspired collaborative attention