Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

2026-09-30

AI Summary

This study addresses the challenge of effectively integrating dispersed evidence in long-horizon clinical reasoning with large language models by proposing MedLoCoMo, a benchmark that systematically evaluates five context strategies—Full, Recent, Episodic, Semantic, and Hybrid—for processing longitudinal patient histories. The findings demonstrate that merely extending context length fails to improve performance; rather, evidence selection and presentation are critical for reliable reasoning. Experiments reveal that Episodic and Hybrid strategies achieve optimal accuracy under long-range dependencies. Furthermore, this work shows that high accuracy does not equate to robust abstention capability, underscoring the necessity of multidimensional evaluation frameworks for assessing the reliability of clinical AI systems.

Abstract

Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning.

Problem

• Longitudinal Clinical Reasoning
• Large Language Models
• Long-context
• Context Strategies
• Abstention

Innovation

• Longitudinal Clinical Reasoning
• Context Strategies
• Large Language Models
• Evidence Retrieval
• Abstention