theorem prover integration

Current benchmarks in formal reasoning and automated theorem proving suffer from incomplete coverage, erroneous annotations, and closed-source code and data—leading to unreliable evaluations, poor reproducibility, and hindered community collaboration. To address these issues, we propose an end-to-end open-source benchmark construction framework: (1) unifying formal and informal statements and proofs; (2) rigorously verifying logical correctness and domain coverage; (3) fully open-sourcing benchmark datasets, evaluation scripts, and baseline models; and (4) systematically identifying and rectifying misleading evaluation practices (e.g., data leakage, undetected overfitting). Our core contribution is the first standardized evaluation suite that simultaneously ensures completeness, verifiability, and openness. This significantly improves result comparability and reproducibility, lowers barriers to entry, and enables fair, cross-method, and cross-community benchmarking and collaborative innovation.

This study addresses the lack of empirical evidence in theorem prover selection by conducting the first systematic, cross-platform comparison of Coq and Idris2—evaluated on a unified task: correctness verification of insertion sort. The methodology employs interactive formal verification, integrating implementation, proof strategy design, and standard library usage to enable both qualitative and empirical analysis across three dimensions: usability, community support, and library ecosystem. Results indicate that Coq exhibits significant advantages in standard library completeness, toolchain maturity, and community resources. In contrast, Idris2 demonstrates innovative potential in proof expressiveness and program-proof integration, leveraging its dependent type system and built-in computational capabilities. This work establishes the first empirically grounded, task-aligned benchmark for cross-prover evaluation and provides practitioners with actionable guidance for formal tool selection and system design.

Current benchmarks for mathematical reasoning predominantly rely on answer matching, which fails to assess the logical correctness of solution processes. This work proposes a hybrid verification pipeline that integrates automated and interactive validation by leveraging structured prompting to guide large language models in generating verifiable solutions. The framework supports both formal and informal reasoning and interfaces with proof assistants such as Lean 4, enabling even small-scale models (≤8B parameters) to participate effectively in collaborative verification. Through a multi-agent architecture and advanced prompt engineering, the approach substantially reduces false positive rates. Experimental results demonstrate high verification accuracy across multiple datasets, and the codebase along with deployment guidelines has been publicly released.

This work addresses the tendency of large language models (LLMs) to produce mathematically plausible yet logically flawed arguments by proposing an intermediate approach between informal reasoning and full formalization. The method guides LLMs to generate typed, compact proof sketches, which are then automatically expanded by a lightweight trusted kernel into explicit proof obligations. These obligations are expressed in a domain-specific language (DSL) to enable efficient verification. By integrating this structured formal scaffolding with the natural expressiveness of LLM-generated reasoning, the approach preserves human-readable argumentation while introducing rigorous formal guarantees. Empirical results demonstrate that this hybrid strategy significantly enhances both the reliability and correctness of mathematical reasoning performed by LLMs.

This work proposes a novel paradigm termed “agent-based proof automation” to address the high cost of manually crafting lengthy formal proof scripts. In this approach, human experts supply key mathematical insights, while large language model (LLM) agents autonomously generate and iteratively refine proof scripts within the Lean 4 environment. Relying solely on off-the-shelf LLMs and lightweight verification tools, the method demonstrates— for the first time—the capacity for efficient, large-scale collaborative formal verification by LLM agents. Evaluated on the 14,000-line System Capless type safety proof, the system successfully completed 189 out of 217 tasks (87% success rate), with only 16% of the tasks requiring human intervention.

This work addresses the ambiguity inherent in natural language reasoning when tackling research-level mathematical problems, which often undermines the reliability of both solution generation and verification. To overcome this limitation, the authors propose a novel collaborative framework that integrates an informal reasoning agent, Rethlas, with a formal verification agent, Archon. Rethlas explores solution strategies, while Archon automatically translates them into machine-checkable proofs in Lean 4. This framework achieves, for the first time, end-to-end automated solving and formal verification of open mathematical conjectures by synergistically combining large language models, theorem retrieval systems (Matlas and LeanSearch), structured task decomposition, iterative refinement, and automated proof synthesis. The approach successfully resolves an open problem in commutative algebra and produces a formally verified proof in Lean 4 with minimal human intervention, establishing a new paradigm for the collaboration between informal and formal mathematical reasoning.

This work proposes a large language model–driven automated theorem proving system that enables human–machine collaborative formal verification. The system employs a Planner–Worker–Verifier multi-agent architecture to decompose proof tasks into parallel subgoals, integrates Lean 4 for automatic formal verification, and manages intermediate reasoning through a shared whiteboard and knowledge base. Innovatively combining agent-based automated proving with interactive user guidance within an open-source framework, it provides a terminal interface to support reproducible collaborative exploration. Experimental results on the ProofNet benchmark demonstrate that the approach significantly outperforms simple baselines. The system is fully open-sourced and designed for reproducible evaluation.

While current large language models can automatically fill proof holes (i.e., eliminate 'sorries') in interactive theorem proving, their generated formalizations often fail expert review due to ill-conceived definitions, insufficiently general theorems, or suboptimal API design. This work presents a semi-autonomous formalization of Grothendieck’s vanishing theorem as a case study and introduces expert review as a central criterion for evaluating the quality of automated formalizations. By integrating large language model assistance, interactive proving, and an iterative refactoring-compression pipeline, the study systematically assesses the high-level design usability of automatically generated content. The findings reveal that measuring success solely by 'sorry' closure is markedly inadequate; expert-driven refactoring substantially improves formalization quality, underscoring the critical role of expert acceptability in evaluating automated formalization efforts.

This work addresses critical reliability issues in existing Lean theorem-proving benchmarks, where inconsistencies between formal statements and informal problem descriptions, along with susceptibility to trivial or adversarial solutions, undermine evaluation validity. To tackle this, the study introduces the first fault taxonomy for formal mathematical datasets and develops an automated auditing toolkit integrating static program analysis, formal verification, semantic auditing via prompt engineering, and manual validation. Applying this framework, the authors conduct a large-scale audit of five prominent benchmarks, uncovering 4,833 issues—including 398 severe defects—and demonstrate that uncorrected flaws significantly distort prover rankings. The paper releases both the auditing tools and corrected dataset snapshots to foster reproducible and trustworthy evaluations in theorem proving.

Existing automated proof synthesis methods struggle with complex theorems in interactive theorem provers and rely heavily on expert knowledge. This work presents the first systematic analysis of failed proof attempts, uncovering critical correlations between human expert proof patterns and successful proofs. Building on these insights, we propose Pattern-Guided Tactic Search (PGTS), a novel approach that integrates deep learning–driven proof synthesis, empirical analysis of proof scripts, and heuristic tactic search guided by expert-derived patterns. Experimental results demonstrate that PGTS improves upon existing tools by proving 8.05% more theorems on standard benchmarks on average and achieves a 20% higher success rate on previously unproven theorems, while also generating more concise proof scripts.