StatEval: A Comprehensive Benchmark for Large Language Models in Statistics

2025-10-10

AI Summary

The statistics domain lacks a systematic, large language model (LLM)-oriented evaluation benchmark. Method: We introduce StatEval—the first comprehensive benchmark for statistical reasoning, covering undergraduate and graduate curricula as well as frontier research topics, comprising 13,817 foundational questions and 2,374 research-level formal proof tasks. We design a scalable multi-agent automated pipeline integrated with human verification to ensure question quality, and develop a fine-grained evaluation framework tailored to statistical computation and formal proof. Contribution/Results: Experiments reveal severe limitations of state-of-the-art LLMs on research-level statistical tasks (e.g., GPT-5-mini achieves <57% accuracy; open-source models perform worse), highlighting the unique challenges of statistical reasoning. StatEval fills a critical gap in the field and provides a rigorous, reliable benchmark for diagnostic model assessment and algorithmic advancement.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Statistical Relational/Logic Learning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have demonstrated remarkable advances in mathematical and logical reasoning, yet statistics, as a distinct and integrative discipline, remains underexplored in benchmarking efforts. To address this gap, we introduce extbf{StatEval}, the first comprehensive benchmark dedicated to statistics, spanning both breadth and depth across difficulty levels. StatEval consists of 13,817 foundational problems covering undergraduate and graduate curricula, together with 2374 research-level proof tasks extracted from leading journals. To construct the benchmark, we design a scalable multi-agent pipeline with human-in-the-loop validation that automates large-scale problem extraction, rewriting, and quality control, while ensuring academic rigor. We further propose a robust evaluation framework tailored to both computational and proof-based tasks, enabling fine-grained assessment of reasoning ability. Experimental results reveal that while closed-source models such as GPT5-mini achieve below 57% on research-level problems, with open-source models performing significantly lower. These findings highlight the unique challenges of statistical reasoning and the limitations of current LLMs. We expect StatEval to serve as a rigorous benchmark for advancing statistical intelligence in large language models. All data and code are available on our web platform: https://stateval.github.io/.

Problem

• Evaluating LLMs' statistical reasoning across foundational and research-level problems
• Addressing the lack of comprehensive benchmarks for statistical intelligence in AI
• Assessing performance gaps in LLMs for computational and proof-based statistical tasks

Innovation

• Automated multi-agent pipeline for problem extraction
• Human-in-the-loop validation ensuring academic rigor
• Tailored evaluation framework for statistical reasoning