llm-assisted theorem proving

Current benchmarks for mathematical reasoning predominantly rely on answer matching, which fails to assess the logical correctness of solution processes. This work proposes a hybrid verification pipeline that integrates automated and interactive validation by leveraging structured prompting to guide large language models in generating verifiable solutions. The framework supports both formal and informal reasoning and interfaces with proof assistants such as Lean 4, enabling even small-scale models (≤8B parameters) to participate effectively in collaborative verification. Through a multi-agent architecture and advanced prompt engineering, the approach substantially reduces false positive rates. Experimental results demonstrate high verification accuracy across multiple datasets, and the codebase along with deployment guidelines has been publicly released.

This work investigates the effectiveness of large language models (LLMs) in assisting interactive theorem proving, particularly within real-world, industrial-scale formal verification tasks. Method: We conduct a systematic evaluation on two authentic verification projects—hs-to-coq and Verdi—using the Rocq proof assistant, employing both quantitative metrics (success rate, proof length, error rate) and qualitative analysis (tactic reasonableness, technical reusability). Contribution/Results: LLMs demonstrate strong capability in generating concise, high-quality formal proofs aligned with classical proof styles, scaling effectively from small to large proofs. Performance critically depends on external dependency information and contextual modeling quality, exhibiting marked heterogeneity across projects. While rare anomalous errors occur, they are substantially mitigated via context enhancement. To our knowledge, this is the first empirical study to characterize LLM capabilities and key limiting factors—such as dependency awareness and context fidelity—in realistic, production-grade formal verification settings, thereby providing foundational evidence and practical guidance for LLM-augmented trustworthy software verification.

This work proposes an end-to-end, large language model (LLM)-driven framework that integrates natural language directly into the formal verification pipeline, addressing the longstanding challenge that existing formal verification methods rely on rigorously defined formal specifications and thus struggle to accommodate safety requirements expressed in natural language. The approach leverages an LLM to automatically translate natural language descriptions into formal safety specifications, which are then used to perform compositional verification of code implementations. By circumventing the traditional dependency on manually crafted formal specifications, the method demonstrates a novel pathway toward bridging informal requirements and rigorous verification. Preliminary experiments indicate its feasibility and potential for enhancing code safety, offering a promising direction for making formal verification more accessible and applicable to real-world software development practices.

This work addresses the poor readability of formal proofs, which hinders comprehension by mathematicians. We propose a structure-aware recursive summarization framework that leverages large language models to generate stepwise, informal, and hierarchical natural-language summaries of formal proofs—e.g., those written in Lean. The method recursively compresses subproofs along the proof dependency graph and then integrates contextual information to produce coherent, natural-language explanations. Crucially, it achieves end-to-end generation of highly readable natural-language proofs while strictly preserving logical fidelity. Experiments on textbook-level theorems and the Lean Mathematical Library demonstrate that the generated summaries match or surpass human-written reference proofs in readability, logical faithfulness, and mathematical rigor. These results validate both the method’s effectiveness and its generalizability across diverse mathematical domains.

This work proposes a novel paradigm termed “agent-based proof automation” to address the high cost of manually crafting lengthy formal proof scripts. In this approach, human experts supply key mathematical insights, while large language model (LLM) agents autonomously generate and iteratively refine proof scripts within the Lean 4 environment. Relying solely on off-the-shelf LLMs and lightweight verification tools, the method demonstrates— for the first time—the capacity for efficient, large-scale collaborative formal verification by LLM agents. Evaluated on the 14,000-line System Capless type safety proof, the system successfully completed 189 out of 217 tasks (87% success rate), with only 16% of the tasks requiring human intervention.

Existing LLM–verifier collaboration frameworks for formal verification lack theoretical guarantees, leading to unstable behavior such as non-termination or divergence. Method: We propose the first formally verified LLM–verifier framework with provable termination and convergence: we model the interaction as a discrete-time Markov chain, establish a quantitative relationship between error-reduction probability δ and expected iteration count, and derive a convergence theorem yielding an analytical upper bound of 4/δ on expected iterations. Contribution/Results: This enables systematic, predictability-driven system design—replacing heuristic tuning with rigorous resource planning. Empirical evaluation across >90,000 tasks demonstrates universal convergence, with measured convergence factor (C_f approx 1.0), confirming tight alignment between theory and practice. The framework provides a quantifiable foundation for resource allocation in safety-critical software verification.

This work addresses the challenges of ensuring correctness and accurately constructing formal specifications when large language models generate code from natural language. We propose a verifiable code generation approach that integrates hierarchical prompting with verification feedback. To support this, we introduce the NL2VC-60 dataset, which leverages Dafny formal specifications and the uDebug platform to prevent vacuous verification. Furthermore, we design a self-repair prompting mechanism guided by structural signatures and verifier feedback. Experimental results demonstrate that our method substantially enhances both verifiability and functional correctness of code generated by open-source large models: Gemma-4-31B achieves a verification success rate of 90.91%, while GPT-OSS-120B improves from 0% to 81.82% under signature-guided prompting, marking the first systematic validation of open-source models’ potential to produce high-assurance code for complex algorithmic tasks.

This study addresses the common omission in current large language model evaluations of code generation—the iterative refinement process inherent in real-world programming and the models’ capacity for self-correction using feedback. The authors propose a novel framework that leverages execution-based feedback, such as compilation errors and test failures, to systematically investigate how reasoning and non-reasoning models utilize such signals across multiple programming languages. Through multidimensional categorization of code failures and extensive cross-model, cross-language experiments, they demonstrate that reasoning models consistently improve over iterations and significantly outperform non-reasoning counterparts. While syntactic and runtime errors prove relatively amenable to correction, logical and algorithmic errors remain challenging, thereby delineating the current limits of feedback-driven repair mechanisms.

Large language models often introduce subtle, hard-to-detect bugs when generating complex software, compromising reliability. This work proposes the first fully automated, project-level code generation and verification framework based on an interactive theorem prover (ITP). The approach separates code with side effects into C++ while formalizing pure logical components in the ITP Rocq, where they are automatically verified and extracted for integration. When proofs fail, the concrete counterexample states guide an LLM agent to autonomously repair the code. In experiments, the system generated 1,859 lines of verified Rocq code and extracted 2,848 lines of C++ within 30 minutes, passing 265 unit tests and 12 hours of AFL++ fuzzing with zero crashes or hangs—outperforming Dafny’s backend, which failed to complete verification under identical conditions.

This work presents the first systematic evaluation of large language models’ (LLMs) ability to infer and repair program loop invariants without auxiliary information. We adopt an empirical framework encompassing diverse open- and closed-source LLMs across multiple scales, integrating domain-knowledge augmentation and few-shot prompting to quantify performance on standard benchmarks for inductive invariant generation and logical defect repair. Results show that LLMs achieve up to 78% success in invariant generation but only 16% in invariant repair—revealing a critical bottleneck in deep logical correction. A key contribution is the identification of auxiliary information—particularly loop semantics prompts and correct examples—as decisive for improving repair accuracy. Our study establishes a reproducible evaluation paradigm for LLM-driven automated program safety analysis and provides concrete, actionable pathways for enhancing invariant repair capabilities.