Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs

2025-10-23

AI Summary

Knowledge Graph Question Answering (KGQA) often fails in complex scenarios involving incomplete knowledge and multi-hop reasoning due to myopic local search and the absence of an external knowledge orchestration mechanism. Method: We propose Graph-RFT, a two-stage reinforcement learning framework that introduces a novel “plan-then-retrieve” paradigm. It explicitly models planning–retrieval actions, employs a coverage-aware multi-reward mechanism, and decomposes questions into subproblems via Cartesian-style factorization—enabling synergistic retrieval from both knowledge graphs and web sources, alongside autonomous reasoning planning. Technically, Graph-RFT integrates chain-of-thought fine-tuning, logical-expression-driven tool invocation, and RL-guided retrieval scheduling. Results: Evaluated on multiple challenging KGQA benchmarks, Graph-RFT significantly improves reasoning path consistency and answer accuracy, empirically validating the efficacy of coordinated planning and retrieval for multi-step reasoning under knowledge incompleteness.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Natural Language Processing: Question Answering
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge Graph Question Answering aims to answer natural language questions by reasoning over structured knowledge graphs. While large language models have advanced KGQA through their strong reasoning capabilities, existing methods continue to struggle to fully exploit both the rich knowledge encoded in KGs and the reasoning capabilities of LLMs, particularly in complex scenarios. They often assume complete KG coverage and lack mechanisms to judge when external information is needed, and their reasoning remains locally myopic, failing to maintain coherent multi-step planning, leading to reasoning failures even when relevant knowledge exists. We propose Graph-RFT, a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm, that enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions. Graph-RFT introduces a chain-of-thought fine-tuning method with a customized plan-retrieval dataset activates structured reasoning and resolves the GRPO cold-start problem. It then introduces a novel plan-retrieval guided reinforcement learning process integrates explicit planning and retrieval actions with a multi-reward design, enabling coverage-aware retrieval scheduling. It employs a Cartesian-inspired planning module to decompose complex questions into ordered subquestions, and logical expression to guide tool invocation for globally consistent multi-step reasoning. This reasoning retrieval process is optimized with a multi-reward combining outcome and retrieval specific signals, enabling the model to learn when and how to combine KG and web retrieval effectively.

Problem

• Enable autonomous planning and adaptive retrieval across knowledge graphs and web sources
• Solve incomplete knowledge coverage and locally myopic reasoning in KGQA
• Develop globally consistent multi-step reasoning with coverage-aware retrieval scheduling

Innovation

• Two-stage reinforcement fine-tuning framework for KGQA
• Plan-retrieval guided RL with multi-reward design
• Cartesian planning module decomposes questions into subquestions