ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

2026-07-30

AI Summary

Current vision-language models suffer from limited performance when processing long visual contexts, primarily due to GPU memory constraints imposed by full-token computation. This work proposes ReToken—a learnable embedding that serves as an explicit retrieval target to efficiently select query-relevant sparse visual tokens from a pre-filled visual key-value cache. Requiring only a single learnable token and leveraging a lightweight architecture with end-to-end fine-tuning, ReToken significantly enhances long-context understanding even with minimal training data. Experiments demonstrate that ReToken improves Qwen3VL-8B and InternVL3.5 by 13.4 and 12.4 points, respectively, on Visual Haystacks, and achieves an 8.0-point gain on zero-shot long-video tasks in LVBench. Both training and inference are feasible on a single H100 GPU.

Abstract

Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Problem

• visual retrieval
• vision-language models
• long visual context
• computational efficiency
• distractors

Innovation

• ReToken
• vision-language models
• visual retrieval
• sparse token selection
• KV cache