multi-hop question design

Existing RAG benchmarks largely overlook query difficulty, leading to inflated and unreliable evaluations. Robust assessment necessitates jointly considering answer quality, response diversity, and query difficulty. Method: We propose a fine-grained difficulty modeling framework based on multi-hop tree structures. Specifically: (1) we design a logically coherent multi-step query synthesis mechanism; (2) we formulate a difficulty metric integrating evidence distribution and reasoning depth; and (3) we establish a controllable data synthesis pipeline enabling difficulty-stratified dataset generation. Contribution/Results: To our knowledge, this is the first work to introduce a difficulty estimation algorithm that jointly evaluates retrieval and generation capabilities within RAG. Experiments show strong correlation (r > 0.85) between our estimated query difficulty and end-to-end RAG performance, significantly enhancing evaluation robustness and interpretability.

This work addresses the lack of a unified analytical framework for retrieval and reasoning pipelines in multi-hop question answering, which has hindered systematic comparison across methods. We propose, for the first time, a four-axis design framework that treats the execution process as the fundamental unit of analysis, encompassing execution plans, index structures, control strategies, and termination criteria. Through a comprehensive literature review and ablation studies, we structurally map prominent approaches—including RAG and agent-based systems—onto this framework using benchmarks such as HotpotQA, revealing consistent trade-offs among effectiveness, efficiency, and evidence faithfulness. Our framework systematically organizes existing design choices, identifies reproducible empirical trends, and highlights key challenges, including structure-aware planning and transferable control strategies.

To address the challenges of poor coordination between reasoning and external knowledge retrieval, as well as limited interpretability in multi-hop question answering, this paper proposes a dynamic knowledge graph construction framework that integrates question decomposition with breadth-first search (BFS)-guided reasoning. The method retrieves and structurally organizes external knowledge on-demand during inference, explicitly generating multi-hop evidence chains to enable synchronous evolution of reasoning paths and their supporting knowledge. Core techniques include retrieval-augmented generation, dynamic knowledge subgraph construction, stepwise question decomposition, and BFS-driven iterative reasoning. Evaluated on MuSiQue, 2WikiMultiHopQA, and HotpotQA, the approach achieves state-of-the-art performance: average exact match (EM) improves by 2.57% and F1 by 2.13%; on HotpotQA specifically, EM and F1 increase by 4.70% and 3.44%, respectively—demonstrating substantial gains in both accuracy and interpretability.

Existing multi-hop question answering methods face a trade-off between parallel fact verification and sequential chain-of-thought reasoning: LLM-based verification excels at parallel processing but struggles to model structured reasoning chains, while KG path retrieval supports chain reasoning yet often introduces redundant paths. Method: We propose DTKG, a dual-track knowledge graph verification and reasoning framework that, for the first time, incorporates cognitive science’s dual-process theory into multi-hop reasoning. DTKG employs a reasoning-pattern classifier to dynamically identify whether a query requires parallel or sequential reasoning, then routes it to either LLM-based semantic verification or KG-based structured path retrieval accordingly. Contribution/Results: This design synergistically combines complementary strengths, substantially reducing redundant retrievals while improving both reasoning efficiency and accuracy. On mainstream multi-hop QA benchmarks, DTKG achieves an average 5.2% absolute accuracy gain over strong baselines and reduces inference latency by 37%, demonstrating robust adaptability and generalizability across heterogeneous reasoning tasks.

This work addresses the challenge of explicitly modeling answer reasoning chains in multi-hop retrieval-augmented generation (RAG) systems under a fixed retrieval budget. The authors propose HKVM-RAG, which introduces a novel key-value separated hypergraph evidence organization mechanism: evidence tuples generated by a large language model form hyperedges serving as retrieval keys, while original passages act as answer values. Rather than replacing dense retrieval, the hypergraph functions as a reusable control signal. The approach integrates weighted hypergraph retrieval, a fixed-basis protocol, and a dense-aware controller fusing ColBERTv2 with HKVM features. Extensive ablation studies at the source level validate its effectiveness. On 2WikiMultiHopQA, MuSiQue, and HotpotQA, HKVM-RAG achieves F1 scores of 88.846, 65.073, and 85.810, respectively—improving upon ColBERTv2 by 11.084, 6.763, and 5.966 points.

Existing question-answering benchmarks primarily emphasize final answer correctness, often neglecting the evaluation of intermediate reasoning processes—particularly the integration of evidence across text, tables, and figures in long, multimodal documents. To address this gap, this work proposes BRIDGE, a multi-hop reasoning benchmark tailored to scientific papers, which introduces fine-grained annotations for multi-hop reasoning paths, supporting both chain-like and fan-out reasoning structures and enabling step-level model diagnostics. Leveraging a multimodal retrieval-augmented generation (RAG) framework combined with large language models, BRIDGE facilitates cross-modal evidence tracing. Experiments reveal that state-of-the-art models exhibit systematic deficiencies in evidence aggregation and cross-modal localization, shortcomings that BRIDGE effectively uncovers—issues otherwise obscured by conventional answer-accuracy metrics.

Existing cross-chart retrieval-augmented generation (RAG) benchmarks are limited by high lexical overlap between queries and evidence and inconsistent reasoning chains, hindering their ability to support complex multimodal analysis. This work proposes ChartWalker, a framework that constructs chart-oriented hierarchical knowledge graphs and employs a structure-aware sampling algorithm to explicitly control query difficulty and granularity, thereby synthesizing high-quality question-answer pairs with multi-hop reasoning paths. Leveraging this approach, we introduce ChartWalker-Bench, the first comprehensive cross-chart RAG benchmark, which exposes significant performance bottlenecks in current state-of-the-art methods. To facilitate future research, we also release ChartWalker-Agent, an open-source agent baseline designed for this challenging task.

Existing Agentic RAG benchmarks lack annotations for intermediate reasoning steps, making it difficult to diagnose failure points in multi-hop retrieval and limiting scalability due to reliance on manual construction. This work proposes the first automated method for constructing an Agentic RAG benchmark with annotated intermediate hops, leveraging large language models to generate 1,305 multi-domain, non-overlapping multi-hop reasoning samples that enable step-by-step verification and fine-grained evaluation. The benchmark introduces hop-level accuracy and reasoning chain completeness metrics, addressing a critical gap in evaluating logical structure consistency. Experiments reveal that even GPT-5 achieves only 22.6% exact match accuracy on the most challenging subset, with primary failures stemming from premature collapse or excessive extension of reasoning chains—highlighting current models’ inability to align with the logical structure of complex tasks.

This work addresses the limitations of existing approaches in multi-hop retrieval-augmented generation, which rely on fixed pipelines and lack dynamic control over evidence manipulation. The authors propose the first unified state-conditioned control framework, modeling multi-hop evidence acquisition as a sequence of atomic operations conditioned on the current reasoning state. A validity filtering layer constructs a feasible action set, from which a learnable controller adaptively selects the optimal operation. Integrating state-conditioned policy learning with the Qwen2.5-7B-Instruct model, the method is optimized end-to-end and achieves F1 scores of 0.5998, 0.5340, and 0.3061 on HotpotQA, 2WikiMultihopQA, and MuSiQue, respectively—significantly outperforming existing controllable baselines. Ablation studies confirm the critical contributions of both the learned controller and the sufficiency-based feedback mechanism.

Current retrieval-augmented generation (RAG) systems exhibit fragility in multi-hop, knowledge-intensive question answering due to implicit reasoning that often leads to retrieval drift and unreliable self-reflection. This work proposes reframing multi-hop RAG as a program synthesis and execution task, wherein reasoning is explicitly modeled through the generation of executable Python programs. Intermediate reasoning states are exposed as program variables, and deterministic feedback from program execution drives self-correction and adaptive retrieval. This approach achieves, for the first time, explicit and verifiable reasoning traces without requiring additional training. It significantly outperforms strong baselines across five benchmarks—including PopQA and HotpotQA—with particularly pronounced gains on compositional multi-hop tasks, and supports both zero-shot and reinforcement learning settings.