Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference

2026-07-29

AI Summary

This work addresses a critical gap in existing visual token pruning evaluations, which focus solely on answer accuracy while neglecting the spatial traceability of OCR-relevant regions—potentially yielding correct answers without local evidential support. To remedy this, the authors propose a training-agnostic auditing framework that introduces, for the first time, a spatial provenance mechanism to jointly assess trade-offs among answer fidelity, spatial origins of visual tokens, intervention efficacy, and computational cost. Through systematic evaluation on Qwen3-VL, LLaVA-1.5, and InternVL3.5 using Target, Random, and Grid token selectors—combined with geometric provenance analysis and detector validation—the study demonstrates that at 30% token retention, the Target strategy achieves an accuracy of 0.786 and significantly higher positive support coverage than Random (0.620 vs. 0.270), while enabling up to 4.32× faster batch prefill and 76.4% peak memory reduction, yet also revealing limited generalizability of task-specific compression.

Abstract

Visual-token pruning is usually judged by answer quality at a fixed retention budget. For text-rich multimodal large language models (MLLMs), this protocol can miss a distinct failure: an answer remains correct even when no retained token is locally traceable to the small OCR region that supports it. We turn this blind spot into an evidence-risk audit that couples answer behavior with geometric token-origin provenance, interventions, and realized cost; transparent training-free selectors isolate controlled operating points. On locked image-disjoint confirmation, Qwen Target at 30% retention has observed accuracy 0.786 versus 0.783 for Full (paired image-cluster difference +0.003, 95% CI [-0.014, +0.020]), yet same-budget Target, Random, and Grid retain sharply different positive-support coverage: 0.620, 0.270, and 0.318. Across Qwen3-VL-8B, LLaVA-1.5-7B, and InternVL3.5-8B, matched controls, interventions, detector tests, and external methods reveal model-specific quality-risk-traceability frontiers that accuracy alone does not expose. Materialized prefixes yield up to 4.32x batch-prefill speedup and 76.4% lower incremental peak memory; full-validation TextVQA and DocVQA further show that favorable target-verification points do not imply task-general compression. Visual-token pruning should therefore report surviving spatial provenance and realized cost alongside quality and compression.

Problem

• visual token pruning
• spatial provenance
• OCR-critical MLLM
• answer traceability
• retention budget

Innovation

• spatial provenance
• visual token pruning
• evidence-risk audit
• OCR-critical MLLM
• traceability frontier