TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics

2026-06-08

AI Summary

Current evaluations of large language models in formal mathematical reasoning are largely confined to competition-style problems, which inadequately capture their capacity for handling complex, highly dependent derivations. To address this limitation, this work introduces TheoremBench—a Lean4-based benchmark comprising nearly one hundred classical theorems—and pioneers a dual-version design featuring both “main theorem” and “premise-augmented” formulations, enabling fine-grained assessment of both complete proofs and intermediate sub-theorem generation. We further propose structured sub-theorem tasks and novel evaluation metrics, including theorem coverage and token efficiency, to establish a multidimensional evaluation framework. Experimental results demonstrate that explicit premises substantially enhance model performance, while also revealing a prevailing tendency among current provers to produce simplistic sub-theorems and verbose tactics rather than concise, efficient proof strategies.

Abstract

LLMs have recently achieved strong results on formal proving benchmarks. However, existing evaluations remain heavily concentrated on competition-style problems and often fail to capture how models behave on longer, more dependency-rich mathematical developments. We introduce TheoremBench, a Lean4 benchmark designed to evaluate theorem provers beyond contest settings. The benchmark is built from nearly one hundred classical theorems and is released in two complementary forms: a plain main version containing one target theorem per instance, and a premised version that expands each theorem into a structured family of related proving tasks consisting of the main theorem together with automatically extracted supporting subtheorems. This design enables evaluation of not only whether the final theorem was proved from scratch, but also of partial progress through the internal proof structure of a theorem. Our experiments show that explicit premises substantially improve performance for Lean4-capable prover models. To provide a comprehensive evaluation, we introduce theorem-level coverage and token-efficiency metrics that expose qualitative differences in proof behavior. The results show that current provers remain strongly biased toward easy subtheorems and often solve theorems through long and inefficient tactic traces rather than compact proof plans. TheoremBench therefore provides a more fine-grained view of formal reasoning ability and highlights the importance of structural benchmark design for evaluating Lean4 theorem provers.

Problem

• theorem proving
• formal mathematics
• benchmarking
• large language models
• Lean4

Innovation

• TheoremBench
• formal theorem proving
• structured proof evaluation
• premised benchmark
• Lean4