RUTA: Principled Visual Token Allocation via Rate-Utility Optimization

2026-08-04

AI Summary

High-resolution images and long videos result in excessively long visual token sequences in vision-language models, leading to substantial computational and memory costs. This work introduces rate–utility trade-offs into visual token compression for the first time and proposes a differentiable, end-to-end dynamic token allocation method. The approach generates candidate tokens conditioned on queries, predicts their retention probabilities via Bernoulli gating, and aggregates information by jointly considering semantic affinity and spatial proximity. By co-optimizing token selection and quantity allocation, the method achieves 88.2%–94.4% of the full-token baseline performance across five benchmarks while using only 2.0%–4.2% of the original visual tokens.

Abstract

High-resolution images and long videos provide vision-language models with rich context for multimodal reasoning and fine-grained perception, but the resulting long visual token sequences make large language model-side computation and memory costly. Existing visual token reducers often operate at prescribed rates, while recent methods adapt token counts across inputs using method-specific learned thresholds or importance predictors. We introduce RUTA, a principled Rate-Utility Token Allocation method that performs pre-LLM reduction by jointly learning which tokens to retain and how many to allocate to each image-query pair. RUTA constructs query-conditioned candidate tokens and predicts a retention probability for each candidate. During training, these probabilities parameterize independent Bernoulli gates, while their sum provides a differentiable training-time estimate of the token count for each pair. Retained tokens serve as anchors that aggregate information from non-retained tokens according to semantic affinity and spatial proximity. RUTA is optimized with a penalized rate-utility objective that balances downstream task loss against expected token usage. Averaged across five benchmarks and measured relative to each backbone's full-token baseline, RUTA uses only $2.0\%$ and $4.2\%$ of visual tokens while preserving $88.2\%$ and $94.4\%$ of task performance on LLaVA-NeXT-7B and Qwen3-VL-8B, respectively.

Problem

• visual token reduction
• rate-utility optimization
• multimodal reasoning
• vision-language models
• token allocation

Innovation

• Rate-Utility Optimization
• Visual Token Allocation
• Vision-Language Models
• Token Reduction
• Differentiable Token Selection