🤖 AI Summary
Existing multimodal large language models (MLLMs) employ static cross-modal tokenization, limiting their ability to emulate human-like, context-sensitive integration of multimodal information. To address this, we introduce— for the first time in MLLMs—the cognitive science principle of *chunking* into tokenizer design, proposing an adaptive cross-modal tokenization framework. Our method comprises three core components: differentiable dynamic boundary learning, hierarchical multi-granularity representation, and vision-language alignment-guided attention. This framework departs from conventional fixed-tokenization paradigms by enabling semantic-driven, context-aware token segmentation. Evaluated on visual question answering (VQA) and complex scene description tasks, our approach achieves absolute improvements of 7.8% and 5.3%, respectively. Moreover, error patterns and attention distributions align significantly more closely with human cognitive behavior. Our work establishes a novel paradigm for developing human-inspired multimodal understanding models.
📝 Abstract
Recent advancements in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in processing diverse data types, yet significant disparities persist between human cognitive processes and computational approaches to multimodal information integration. This research presents a systematic investigation into the parallels between human cross-modal chunking mechanisms and token representation methodologies in MLLMs. Through empirical studies comparing human performance patterns with model behaviors across visual-linguistic tasks, we demonstrate that conventional static tokenization schemes fundamentally constrain current models' capacity to simulate the dynamic, context-sensitive nature of human information processing. We propose a novel framework for dynamic cross-modal tokenization that incorporates adaptive boundaries, hierarchical representations, and alignment mechanisms grounded in cognitive science principles. Quantitative evaluations demonstrate that our approach yields statistically significant improvements over state-of-the-art models on benchmark tasks (+7.8% on Visual Question Answering, +5.3% on Complex Scene Description) while exhibiting more human-aligned error patterns and attention distributions. These findings contribute to the theoretical understanding of the relationship between human cognition and artificial intelligence, while providing empirical evidence for developing more cognitively plausible AI systems.