WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning

2026-06-10

AI Summary

Current language model agents often rely on memorized content, fabricated evidence, or unfounded causal reasoning when predicting real-world events, making it difficult to assess their genuine reasoning capabilities. This work proposes the first triaxial evaluation framework that jointly measures outcome accuracy, evidence quality, and reasoning plausibility. By restricting information access within defined temporal boundaries, the framework enables multidimensional scoring of agents’ probabilistic forecasts, cited evidence, and post-hoc causal graphs. We develop an agent-based automated pipeline to construct a benchmark of prediction tasks annotated with timestamped evidence and reference causal graphs. Evaluation across 345 tasks demonstrates that time-constrained retrieval significantly improves prediction accuracy and that causal graphs aid in identifying pivotal events; however, agents still struggle to translate reliable evidence into well-calibrated probabilistic predictions.

Abstract

Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether agents genuinely forecast requires more than final-answer accuracy: a model may be correct by recalling memorized training facts, citing fabricated evidence, or producing an unsupported causal story. We present WorldReasoner, an evaluation framework for temporally valid event forecasting. Each task gives an agent a resolved forecasting question, a simulated forecast date, and access only to evidence available before that date; after resolution, the framework scores the submitted probability, cited evidence, and optional causal event graph. WorldReasoner reports three complementary axes: outcome quality against resolved answers, evidence quality over cited sources, and reasoning quality against post-resolution hindsight graphs. The benchmark is built by an agentic construction pipeline that generates forecasting questions, collects time-stamped evidence, and builds hindsight reference graphs at scale, yielding 345 resolved tasks derived from 14,141 articles with graphs covering 8,087 extracted events. Across six controlled agent settings, temporally valid retrieval is the strongest driver of outcome accuracy; causal graph construction improves key-event recovery; and correct graph-enabled forecasts are more strongly grounded in key events and relevant sources, yet agents still struggle to convert grounded evidence into calibrated probabilities.

Problem

• event forecasting
• language model agents
• reasoning evaluation
• temporal validity
• causal reasoning

Innovation

• temporally valid forecasting
• causal event graph
• evidence grounding
• reasoning evaluation
• language model agents