MM-ContextFold: Context Folding for Multimodal Agentic Retrieval

2026-09-19

AI Summary

为解决多模态代理检索中的上下文爆炸问题，MM-ContextFold通过仅在需要时加载原始图像，并将图像相关子任务结果文本化后整合回主上下文的方法，提高了任务准确性和效率。

Abstract

Multimodal Agentic Retrieval (MAR) requires agents to solve complex information-seeking tasks by iteratively invoking external tools. Typical frameworks such as ReAct maintain raw multimodal inputs and the accumulating interaction history in a single, ever-growing context, leading to the context explosion problem. While existing methods alleviate this issue by compressing redundant text, effective strategies for managing token-intensive visual content remain largely underexplored. To address this gap, we first conduct a systematic empirical study of approximately 10,000 trajectories. The results show that as visual cues are progressively extracted through external tools and textualized into the context, raw images become increasingly redundant. Continued image retention is associated with higher output entropy and can even degrade task accuracy. Motivated by these findings, we propose MM-ContextFold, a training-free framework that loads raw images only when needed. It maintains a persistent, text-only main context for high-level planning and spawns ephemeral branch contexts for image-dependent subtasks. Within each branch, the agent loads the relevant images, completes the subtask, and folds the result back into the main context as a concise textual summary; the images and branch trace are then discarded. Experiments on seven MAR benchmarks across five backbone models show that MM-ContextFold improves average accuracy by 6.3 percentage points over ReAct while reducing the working context length by 27.5\%.

Problem

• Multimodal Agentic Retrieval
• context explosion
• visual content management

Innovation

• MM-ContextFold
• context folding
• multimodal agentic retrieval
• token-intensive visual content management
• ephemeral branch contexts