🤖 AI Summary
This work introduces the first mathematical proof tutoring system that achieves closed-loop integration of large language models (LLMs) with the Lean theorem prover, addressing critical pedagogical gaps in formal proof education—including natural-language interaction, error localization, correct step generation, and explanatory feedback. Methodologically, it combines automated formalization, Lean compilation-based verification, LLM-driven candidate tactic generation, heuristic proof search, and pedagogically-aware natural-language feedback generation grounded in formal proof context. Its core contribution is a bidirectional LLM–Lean collaboration: Lean ensures formal correctness and reliability, while the LLM enables real-time, instructionally grounded error correction and intuitive explanation. Evaluated on PeanoBench, the system achieves a 57% formalization rate for correct proof steps and a 30% error-step localization rate; its natural-language prompts significantly outperform existing baselines in both accuracy and relevance.
📝 Abstract
We present LeanTutor, a Large Language Model (LLM)-based tutoring system for math proofs. LeanTutor interacts with the student in natural language, formally verifies student-written math proofs in Lean, generates correct next steps, and provides the appropriate instructional guidance. LeanTutor is composed of three modules: (i) an autoformalizer/proof-checker, (ii) a next-step generator, and (iii) a natural language feedback generator. The first module faithfully autoformalizes student proofs into Lean and verifies proof accuracy via successful code compilation. If the proof has an error, the incorrect step is identified. The next-step generator module outputs a valid next Lean tactic for incorrect proofs via LLM-based candidate generation and proof search. The feedback generator module leverages Lean data to produce a pedagogically-motivated natural language hint for the student user. To evaluate our system, we introduce PeanoBench, a human-written dataset derived from the Natural Numbers Game, consisting of 371 Peano Arithmetic proofs, where each natural language proof step is paired with the corresponding logically equivalent tactic in Lean. The Autoformalizer correctly formalizes 57% of tactics in correct proofs and accurately identifies the incorrect step in 30% of incorrect proofs. In generating natural language hints for erroneous proofs, LeanTutor outperforms a simple baseline on accuracy and relevance metrics.