CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference

2026-07-15

AI Summary

This work addresses the high computational cost and latency of large vision-language models (LVLMs) caused by processing excessive visual tokens. Existing pruning methods either neglect textual semantics or operate inefficiently within the large language model (LLM). To overcome these limitations, the authors propose CRISP, a text-driven, two-stage visual token pruning framework applied prior to LLM inference. The first stage aligns user instructions with visual tokens to retain task-relevant evidence, while the second enhances scene context completeness through semantic diversity. CRISP is the first approach to enable efficient, text-aware pruning before the LLM, achieving 99.5% of the original accuracy on LLaVA-1.5 and LLaVA-NeXT using only about half the visual tokens, thereby reducing both inference cost and latency by more than twofold.

Abstract

Large Vision-Language Models (LVLMs) typically require processing hundreds to thousands of visual tokens, leading to substantial inference overhead. Existing visual token pruning methods either operate before the LLM using text-agnostic heuristics or prune inside the LLM at the cost of efficiency and noisy cross-modal attention. To address these limitations, we propose CRISP, a pre-LLM yet text-driven visual token pruning framework that preserves both instruction-relevant evidence and essential scene context. CRISP works in a two-stage pipeline: Stage 1 first identifies text-aligned visual tokens, and Stage 2 enhances contextual completeness through semantic diversity. Extensive experiments on LLaVA-1.5 and LLaVA-NeXT demonstrate that CRISP achieves superior performance retention under aggressive pruning ratios, maintaining up to 99.5% accuracy while reducing inference cost and latency by more than 2 times. CRISP serves as a practical solution for efficient LVLM inference, especially in resource-constrained scenarios.

Problem

• visual token pruning
• efficient LVLM inference
• text-driven pruning
• inference overhead
• cross-modal attention

Innovation

• visual token pruning
• text-driven
• pre-LLM
• semantic diversity
• efficient LVLM inference