RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs

2026-08-07

AI Summary

This work addresses the high computational cost of long visual token sequences in multimodal large language models, which burdens prefilling and KV cache usage, while existing training-free pruning methods lack explicit modeling of object region coverage. The authors formalize visual token pruning as a role-oriented regional evidence allocation problem, partitioning tokens under a fixed budget into three categories: semantic cores, complementary context, and fine-grained details. They introduce Attention Anchor Regions (AARs) as lightweight object proxies to guide differentiated retention. Leveraging techniques such as text-conditioned attention calibration, position- and prompt-guided object priors, and redundancy filtering, the proposed method significantly outperforms existing approaches on LLaVA and Qwen-VL families—achieving 96.5% of original performance at an 88.9% pruning rate on LLaVA-1.5 and accelerating inference by 1.33×.

Abstract

Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive. Existing training-free pruning methods select tokens by importance, diversity, or spatial coverage, but treat retained tokens as interchangeable and do not explicitly track which object-related regions are already covered. We present RoRA, a training-free framework that casts visual token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions tokens into a protected semantic core, complementary context, and fine-grained detail. It first calibrates text-conditioned attention with a positional prior and a prompt-calibrated object prior, then builds Attention-Anchored Regions (AARs) from high-confidence anchors as lightweight proxies for covered object support. Context is explored mainly outside AARs, while a small AAR-guided budget restores local detail; pairwise similarity is used only for context-stage redundancy filtering. Under matched budgets, RoRA consistently outperforms strong training-free baselines across LLaVA and Qwen-VL families, retaining most of the unpruned accuracy even at aggressive pruning ratios, e.g., 96.5% of full performance at 88.9% pruning on LLaVA-1.5, and improving over D2Pruner by about 5% on Qwen3-VL at 75-90% pruning. At a 66.7% pruning ratio, RoRA requires only 0.7 ms for token selection and reduces end-to-end inference time by 24.6%, corresponding to a 1.33x speedup over unpruned inference on an NVIDIA H800.

Problem

• visual token pruning
• multimodal large language models
• KV-cache storage
• object-related regions
• training-free pruning

Innovation

• visual token pruning
• role-oriented allocation
• attention-anchored regions
• training-free MLLM acceleration
• multimodal large language models