TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answering

2026-07-21

AI Summary

This work addresses the limitations of existing retrieval-augmented generation (RAG) systems in long-document multimodal question answering, which typically employ query-agnostic evidence utilization strategies and thus fail to adapt to the varying demands for textual, visual, or structured evidence across different questions. To overcome this, the authors propose TAP-RAG, a novel framework featuring a Task-Aware Policy Controller (TAPC) that dynamically predicts task types and orchestrates multimodal evidence accordingly. Specifically, Task-Aware Query-Guided Flow Diffusion (TA-QFD) processes textual and structural information from document graphs, while Task-Aware Visual Enhancement (TAVE) selectively retrieves relevant page images. The framework further integrates multi-source evidence to support abstention when answer confidence is low. Evaluated on DocBench and MMLongBench-Doc, TAP-RAG achieves state-of-the-art accuracy of 70.2 and 46.7, respectively, outperforming strong baselines by 9.1 and 4.5 percentage points.

Abstract

Long-document multimodal question answering requires more than retrieving relevant chunks from a large document. Different queries require different evidence behavior. Existing multimodal RAG systems improve evidence access through text chunks, page images, graph links, or heterogeneous document elements, but they often apply a largely query-agnostic evidence-use strategy. We present TAP-RAG, a task-aware policy-controlled RAG framework for long-document multimodal QA. TAP-RAG contains a main controller, the Task-Aware Policy Controller (TAPC), and two policy-guided evidence executors: Task-Aware Query-Guided Flow Diffusion (TA-QFD) and Task-Aware Visual Enhancement (TAVE). For each query, TAPC predicts the task prior, estimates visual/local/global evidence signals, and produces an executable policy. TA-QFD then expands textual and structural evidence over the multimodal document graph, while TAVE selectively inspects page images when visual or layout evidence is needed. A guarded synthesis stage fuses text, visual, and structural evidence and abstains when support is insufficient. On DocBench and MMLongBench-Doc, TAP-RAG achieves the best overall accuracy among the compared systems, improving over a matched multimodal-RAG baseline by +9.1 points (61.1 to 70.2) and +4.5 points (42.2 to 46.7), respectively.

Problem

• long-document multimodal question answering
• evidence-use strategy
• query-agnostic
• multimodal RAG
• task-aware policy

Innovation

• Task-Aware Policy Control
• Multimodal RAG
• Evidence Fusion
• Long-Document QA
• Visual Enhancement