SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models.

2026-09-17

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Multimodal large language models face inference efficiency bottlenecks arising from dual redundancy in data and computation. This work proposes SPIDER, a training-free acceleration framework that achieves fine-grained optimization through multi-layer semantic visual token pruning and an adaptive sub-layer skipping mechanism. Specifically, the method uncovers the shifting patterns of semantic focus across intermediate layers and quantifies the differential contributions of attention and feed-forward network modules to guide dynamic computational allocation. Evaluated on LLaVA-NeXT-7B, SPIDER reduces FLOPs by 79% while retaining 96% of baseline performance. Furthermore, it demonstrates broad compatibility with various mainstream architectures without requiring any fine-tuning.

Abstract

Multimodal Large Language Models face significant efficiency challenges that stem from two distinct yet coupled sources: data redundancy and computational redundancy. While most methods focus on data redundancy by pruning visual tokens from the output of the visual encoder or computing redundancy in LLM decoders using blockwise importance, the finer-grained inter-layer representation shifts and the distribution differences within the layers themselves have not been fully explored. In this work, we comprehensively investigate this dual-level inefficiency. We posit that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away. Furthermore, we reveal the differential contributions of Attention and FFNs across distinct LLM decoder layers. Building upon these discoveries, we propose SPIDER, a training-free framework that integrates multi-layer Semantic visual token PrunIng with an aDaptive sub-layER skipping mechanism. Experimental evaluations demonstrate that SPIDER consistently maintains strong performance across various MLLM architectures and reduction ratios. For instance, on LLaVA-NeXT-7B, SPIDER reduces FLOPs by $80\%$ while maintaining 96$\%$ of the baseline performance.

Problem

• Multimodal Large Language Models
• Visual Token Pruning
• Data Redundancy
• Computational Redundancy
• Efficiency

Innovation

• Multimodal Large Language Models
• Visual Token Pruning
• Adaptive Sub-Layer Skipping
• Training-Free
• Computational Efficiency