FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents

2026-07-21

AI Summary

Existing AI agents exhibit inconsistent performance in understanding and reasoning over real-world, complex financial documents, largely due to the absence of evaluation benchmarks that reflect industrial settings. This work proposes FinanceComplexQA—the first open-ended, generative benchmark specifically designed for complex-layout financial documents—encompassing six realistic scenarios, seven task types, and 2,026 challenging questions. The benchmark leverages a novel Finance-LaTeX SKILL pipeline to synthetically generate 2,000 professional documents and 6,000 bilingual question-answer pairs. Integrated with RAG, multi-hop reasoning, and an Agent-as-a-Judge evaluation framework, FinanceComplexQA enables systematic assessment of mainstream agents across critical dimensions such as numerical computation, summarization, and domain-specific analysis, thereby uncovering their strengths and limitations in practical financial contexts.

Abstract

Agentic Reasoning has become a transformative force in financial analysis due to its ability to integrate large-scale information and generate reliable and accurate content. However, when handling complex real-world problems, different agents still show significant performance variation. In this work, we design Finance-LaTeX SKILL, a skill for synthesizing financial documents with complex layouts based on expert knowledge. Using an agent workflow built on this skill, we generate 2,000 professional financial documents along with 6,000 high-quality question-answer pairs. To evaluate the overall capability of agents, we introduce FinanceComplexQA, a comprehensive open-ended generation benchmark for financial documents that closely resembles real-world scenarios. It contains 2,026 deep research tasks targeting 1009 financial documents. FinanceComplexQA has 8 key features: bilingual support; coverage of six mainstream scenarios and seven tasks; expert-level document reasoning questions; deep research of complex layouts; relatively stable and permanent reference answers; and precise evaluation through an Agent-as-a-Judge with multiple evaluation metrics. Using FinanceComplexQA, we conduct a comprehensive evaluation of leading RAG systems and agentic reasoning tools for financial document QA. Through identifying and analyzing failure cases, we provide an in-depth study of their capabilities in numerical computation, multi-hop reasoning, content summarization, and industry analysis.

Problem

• Agentic Reasoning
• Financial Documents
• Complex QA
• Benchmark
• Open-ended Generation

Innovation

• Agentic Reasoning
• Financial Document Generation
• Complex Layout Synthesis
• FinanceComplexQA Benchmark
• Agent-as-a-Judge