When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware

2026-08-04

AI Summary

This work demonstrates that visual token compression does not necessarily reduce end-to-end latency, as its acceleration efficacy depends critically on the pruning strategy and system context. The authors propose a stage-level latency decomposition framework that systematically evaluates the trade-offs of pre-visual and post-visual pruning by accounting for decision overhead, shared computation, and skippable operators. Leveraging a lightweight predictor, image-size heuristics, autoregressive probes, and Holm–Bonferroni multiple testing correction, empirical results on RTX 3090 and A100 GPUs show that pre-visual routing—by skipping redundant preprocessing and encoding stages—significantly outperforms post-visual approaches. Notably, on the A100, pre-visual routing achieves greater speedup than a post-visual strategy with 8× token compression, with both methods exhibiting significantly negative latency confidence intervals.

Abstract

Fewer visual tokens do not guarantee lower end-to-end latency. We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid. A stage-level decomposition reconciles these components with measured end-to-end latency. In a 30-example pilot, the two tested autoregressive probes remain slower than Full despite state reuse. A lightweight post-vision predictor yields paired confidence intervals below zero on RTX 3090 and A100 and remains significant after a conservative all-pairs Holm correction. A pre-vision image-size rule also yields intervals below zero on both GPUs, although neither comparison remains significant after the same correction. Pre-vision routing has a structural opportunity unavailable to post-vision pruning: it can avoid preprocessing and vision encoding. On A100, this opportunity outweighs a nearly eightfold larger downstream token reduction by the post-vision policy. Reported quality is conditional on examples answered correctly by Full and is not benchmark accuracy.

Problem

• visual tokens
• multimodal inference
• end-to-end latency
• decision overhead
• token reduction

Innovation

• visual token reduction
• multimodal inference acceleration
• break-even analysis
• pre-vision routing
• stage-level latency decomposition