Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models

2026-09-20

AI Summary

为解决多模态大语言模型中视觉token修剪导致的感知和定位能力下降问题，提出了一种层感知的位置嵌入策略。

Abstract

Multimodal large language models (MLLMs) incur substantial computational overhead due to the reliance on hundreds of visual tokens to represent images. While token pruning has emerged as a promising approach to reduce the inference cost of MLLMs, existing methods typically reassign position embeddings to the retained tokens using either sparse or continuous position embeddings, each introducing distinct limitations. Sparse position embeddings tend to decrease the attention value allocated to visual tokens, thereby degrading the perception capability of MLLMs, whereas continuous position embeddings disrupt the original spatial correspondence of visual tokens, leading to weakened grounding capability. To mitigate this issue, we perform layer-wise analysis of the language decoder and observe that intermediate layers play a critical role for maintaining the grounding capability of MLLMs under token pruning. Based on this observation, we propose a layer-aware position embedding strategy, which switches to sparse position embeddings at grounding-sensitive layers while maintaining continuous position embeddings elsewhere. Extensive experiments across representative pruning methods and diverse benchmarks demonstrate that our approach improves the comprehensive multimodal performance of pruned MLLMs compared with standard sparse and continuous position embeddings.

Problem

• visual tokens
• position embeddings
• multimodal large language models
• token pruning
• grounding capability

Innovation

• Layer-Aware Position Embeddings
• Visual Token Pruning
• Multimodal Large Language Models
• Grounding Capability