Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

2026-06-08

AI Summary

Although large language models can accurately memorize factual knowledge, they struggle to perform implicit multi-hop reasoning within a single forward pass. This work constructs a rigorously isolated natural language experimental setting and demonstrates, for the first time, that exposure to compositional contexts during pretraining is a necessary condition for such reasoning—and that this capability does not generalize to unseen entities. Through controlled data construction, multi-hop question answering evaluation, and nine data augmentation strategies, the study reveals that even when single-hop accuracy reaches 97%, models still fail significantly on multi-hop tasks. Moreover, data augmentation improves reasoning performance only when the pretraining corpus includes relevant compositional contexts.

Abstract

Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born?" and "Who is $Y$'s closest friend?" correctly but fails on "When was $Y$'s closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning.

Problem

• Multi-Hop Reasoning
• Pretraining Exposure
• Knowledge Composition
• Large Language Models
• Compositional Generalization

Innovation

• multi-hop reasoning
• pretraining exposure
• compositional generalization
• knowledge composition
• data augmentation