Search-on-Graph-R1: Training Large Language Models to Search Knowledge Graphs with Reinforcement Learning

2026-07-20

AI Summary

This work addresses the challenge of efficient multi-hop reasoning in knowledge graph question answering, where existing large language model approaches often rely on costly external tools or auxiliary modules. The authors propose a method that internalizes navigation capabilities directly into an 8B-parameter language model through supervised fine-tuning and reinforcement learning. High-quality training trajectories are generated using a teacher model combined with ground-truth SPARQL queries. Notably, this approach achieves state-of-the-art performance without requiring any auxiliary modules or LLM-based verifiers, outperforming frozen, larger-scale models. The framework demonstrates strong transferability across model families and establishes new best results on WebQSP, CWQ, and GrailQA—reaching the current peak on CWQ—while relying solely on the model itself during inference.

Abstract

Knowledge graph question answering (KGQA) requires navigating from topic entities to an answer several relations away. Recent methods prompt a frontier LLM to explore the graph through a retrieval tool, but their reliance on frontier-scale inference makes them costly to deploy. We present Search-on-Graph-R1 (\sogrone{}), which internalizes this navigation into a compact 8B model through supervised fine-tuning (SFT) followed by reinforcement learning (RL). Our central idea is to scaffold a frontier teacher with each question's gold SPARQL query, so the teacher traverses a known answer-bearing path with a live \texttt{Search} tool rather than having to discover the path itself. Since every call executes against a live Freebase server, the resulting trajectories are grounded in the knowledge graph by construction. On WebQSP, CWQ, and GrailQA, \sogrone{} at 8B surpasses every frozen frontier-LLM system in our comparison and posts the strongest results on CWQ of any system we compare against. It does so using no auxiliary module at inference and no LLM judge during training. Isolating each training stage shows that SFT and RL contribute complementary gains, our approach transfers across model families, and RL learns to reach answers in fewer \texttt{Search} calls than its SFT initialization.

Problem

• Knowledge Graph Question Answering
• Multi-hop Reasoning
• Large Language Models
• Graph Navigation
• Cost-efficient Inference

Innovation

• Knowledge Graph Question Answering
• Reinforcement Learning
• Supervised Fine-Tuning
• Compact Language Model
• Multi-hop Reasoning