Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

2026-07-17

AI Summary

This work addresses a critical gap in existing evaluations: the lack of assessment of large language models’ ability to autonomously formulate testable hypotheses prior to drawing conclusions. To bridge this gap, the authors introduce the Prospective Hypothesis Discovery (PHD) task, which requires models to generate grounded, discriminative, and verifiable hypotheses from incomplete or anomalous evidence. They further present HypoArena, the first formalized benchmark for PHD, comprising a large-scale pre-conclusion dataset constructed via Retrospective Context Regression, and an integrated Forge–Audit evaluation pipeline. HypoArena combines the Bradley–Terry–Davidson ranking algorithm with a six-dimensional human scoring rubric to enable collaborative automated and human assessment of open-ended hypotheses. Experiments across 15 state-of-the-art models demonstrate that HypoArena effectively differentiates model capabilities, yielding rankings highly aligned with human experts and substantially outperforming conventional absolute scoring methods.

Abstract

Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured. We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discriminative, and testable hypothesis spaces from inconclusive evidence, including anomalous observations and fragmented records, to guide subsequent investigation. To evaluate this capability, we introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets. To construct HypoData at scale, we propose Retrospective Context Regression, a Forge--Audit pipeline that reconstructs pre-conclusion contexts from completed expert documents by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate. Because PHD admits multiple valid outputs, HypoEval combines bidirectional pairwise judgments with Bradley--Terry--Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. Experiments on 15 frontier LLMs reveal clear capability stratification and model-dependent effects of structured analytical skills, with gains for several lower-performing models on HypoArena but regressions for other systems, including a top-performing model. Compared with absolute rubric scoring, arena evaluation resolves finer-grained differences among models, with aggregated rankings showing strong agreement with human experts and an independent judge. Together, these results support treating PHD as a distinct target for evaluating how LLMs formulate investigative directions when final conclusions are withheld. Our code and data are publicly available at github.com/SKYLENAGE-AI/HypoArena and github.com/SKYLENAGE-AI/HypoArena.

Problem

• Prospective Hypothesis Discovery
• Large Language Models
• Hypothesis Generation
• Scientific Discovery
• Open-ended Reasoning

Innovation

• Prospective Hypothesis Discovery
• HypoArena
• Retrospective Context Regression
• Open-ended Evaluation
• Hypothesis Generation