Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

2026-08-07

AI Summary

This work addresses the challenge that large language models often produce unreliable statistical conclusions in automated hypothesis testing due to subtle reasoning errors, a limitation exacerbated by the absence of benchmarks evaluating statistical validity. To bridge this gap, the authors introduce P-Bench—the first benchmark specifically designed to assess statistical validity in open-ended hypothesis testing—and develop Fisher-R1-14B, an open-source agent trained via synthetic data generation and reinforcement learning with verification-based rewards. Experimental results demonstrate that Fisher-R1 achieves a 21% higher single-trial success rate than DeepSeek-V4-Pro on P-Bench, with gains reaching 26% on the most challenging tasks, significantly outperforming mainstream baselines including GPT-5.4. These findings validate Fisher-R1’s capacity for reliable statistical reasoning in real-world scientific scenarios.

Abstract

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Problem

• hypothesis testing
• statistical reasoning
• LLM agents
• p-value validity
• reliability

Innovation

• hypothesis testing
• reinforcement learning
• statistical reasoning
• benchmarking
• LLM agents