Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

2026-08-04

AI Summary

This work addresses the high computational overhead of visual token processing in multimodal large language models, where existing pruning methods rely on fixed intermediate-layer attention maps, leading to poor adaptability across samples and excessive computation. To overcome these limitations, the authors propose MAP, a novel sample-adaptive pruning framework that dynamically selects the optimal intermediate layer for each input via a question-aware teacher model and trains a lightweight predictor to estimate visual token importance directly from the input—enabling pre-computation pruning without actual attention calculation. Integrating attention distillation with a diversity-aware pruning criterion, MAP retains only 5.56% of visual tokens on LLaVA-NeXT-7B while preserving 97.5% of the original performance, achieving a 3.09× end-to-end speedup.

Abstract

Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens. Visual token pruning can reduce this cost, but requires accurate token importance estimates. Recent studies have demonstrated that text-to-vision attention from middle language model layers can effectively guide visual token pruning, typically using attention from a predefined middle layer to select the visual tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting attention under the original and reference questions, and distills attention from the selected layer into a lightweight predictor that estimates visual token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer. Thus, MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques. Across ten benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual tokens, yielding a 3.09x end-to-end speedup.

Problem

• visual token pruning
• multimodal large language models
• middle-layer attention
• token importance estimation
• inference efficiency

Innovation

• visual token pruning
• middle-layer attention prediction
• question contrastive teacher selection
• multimodal large language models
• attention distillation