The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents

2026-08-06

AI Summary

This study addresses the challenge that large language models often fail in multi-hour long-horizon tasks due to a “horizon gap,” leading to forgetting early decisions, premature termination, or goal drift. Through a systematic review of 1,547 papers, it introduces the first cross-classification framework based on task lifecycle phases—planning, memory, execution, training, and evaluation—and the locus of information representation. This framework disentangles the commonly conflated notions of task length, context window size, and long-term memory, while highlighting the critical role of process signals. By integrating systematic data collection, leakage filtering, and cross-dimensional categorization with process rewards, credit assignment, and trajectory diagnostics, the work demonstrates that reliance solely on outcome signals inevitably fails as task horizons extend. It concludes by identifying three key open problems: capability decoupling, bias management in process signals, and the development of reliability theory for long-horizon tasks.

Abstract

Frontier language models solve reasoning problems in a single forward pass that would have been research contributions years ago, yet fail at multi-hour tasks: losing track of earlier decisions, declaring half-finished work done, or drifting from goals. We call this the horizon gap and survey 1,547 arXiv papers (2024-2026) collected via systematic seed harvest with a disclosed 26.8% bleed filter, extended by targeted supplementation. We disambiguate three routinely conflated properties: long-horizon (task property: required steps), long-context (model property: token capacity), and long-term memory (system property: persistence across steps/sessions). We organize the corpus into six categories tracking a long-horizon task's lifecycle -- planning, memory, execution, training, evaluation, and foundations/safety -- crossed with an axis capturing where horizons are carried (within-context, within-task-beyond-context, or cross-task-persistent). Across all categories, we find the same pattern: outcome-only signals grow uninformative as horizons lengthen, and the field's response -- whether process reward models, credit assignment, or trajectory-level diagnostics -- manufactures denser step-level signals. We treat critical and diagnostic literature as first-class threads throughout, arguing that segregating critique from method would routinely split single papers across chapters. We close by naming open measurement problems: decomposing model versus harness capability, managing correlated bias in process-level signals used for both training and evaluation, and whether long-horizon reliability admits general predictive theory.

Problem

• horizon gap
• long-horizon tasks
• large language models
• task execution
• reasoning reliability

Innovation

• horizon gap
• long-horizon agents
• process-level signals
• systematic literature review
• long-term memory