Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding

2026-07-27

AI Summary

Existing approaches struggle to jointly achieve precise visual grounding and complex semantic reasoning, primarily due to the limited expressiveness of textual coordinate prediction and the absence of spatial reference in implicit tokens. This work proposes Mixture-of-Thought-Tokens (Motto), a novel framework that introduces spatially explicit thought tokens to enable visually interpretable perception and constructs context-adaptive dynamic reasoning chains. For the first time, Motto unifies and jointly optimizes dense grounding and high-order reasoning within a single architecture. The method integrates spatially anchored tokenization with fine-tuning of multimodal large language models and demonstrates state-of-the-art performance across diverse free-form referring tasks on the newly introduced PR-Bench benchmark, significantly narrowing the capability gap between perception and reasoning.

Abstract

Multimodal Large Language Models have made great progress in grounding tasks, yet existing methods still struggle to unify precise localization and complex reasoning. For one thing, text-based methods rely on coordinates or index prediction, severely limiting the perceptual capabilities of the model for dense visual objects. Meanwhile, latent token-based methods employ special tokens without inherent spatial references and use a decoding mechanism that lacks thinking steps, weakening high-level reasoning capabilities. Consequently, developing a unified framework that excels in both perception and reasoning remains challenging. To address this, we propose Mixture-of-Thought-Tokens (Motto), a new free-form multimodal grounding method that bridges the perception-reasoning gap, enabling MLLMs to empower diverse, arbitrary grounding queries. Specifically, we introduce Spatially-Grounded Thought Tokenization to explicitly align special tokens with spatial locations for clear spatial correspondence and visual interpretability. We further design a Context-Adaptive Chain-of-Tokens that dynamically switch grounding modes within an interleaved reasoning chain, achieving robust grounding across tasks of varying complexity. In addition, we construct PR-Bench, a new referring expression comprehension benchmark to evaluate the perception-reasoning gap. Extensive experiments demonstrate that Motto achieves state-of-the-art performance across diverse free-form grounding tasks.

Problem

• multimodal grounding
• perception-reasoning gap
• spatial localization
• complex reasoning
• referring expression comprehension

Innovation

• Mixture-of-Thought-Tokens
• Spatially-Grounded Thought Tokenization
• Context-Adaptive Chain-of-Tokens
• multimodal grounding
• perception-reasoning unification