🤖 AI Summary
This work addresses the susceptibility of large language models to “context contamination” when verifying complex mathematical proofs, which often masks logical errors. The authors propose a step-level verification framework that preserves the full context of each inference step and strictly restricts the set of admissible theorems for reference, enabling fine-grained validation of research-level proofs. By integrating LLM-guided reasoning constraints, an adversarial benchmark (FirstProof Challenge), and systematic ablation studies, the method substantially outperforms conventional global evaluation approaches in precisely identifying subtle logical flaws. Remaining misjudgments predominantly stem not from severe hallucinations but from “over-rigor”—arising when domain-specific conventions are left implicit—thereby exposing latent ambiguities in existing expert benchmarks and advancing a more human-like, cautious paradigm for mathematical verification.
📝 Abstract
Large Language Models (LLMs) struggle to rigorously verify complex mathematical proofs. Standard global evaluation approaches suffer from "context poisoning," in which superficially plausible statements mask subtle logical flaws, leading to hallucination or over-skepticism. To address this, we shift from global evaluation to strict step-level verification: our framework maintains detailed context for each deduction step and strictly constrains the sources of applied theorems. We evaluate on a carefully curated adversarial diagnostic suite of research-level proofs drawn from the FirstProof challenge. A systematic ablation study demonstrates that these deductive constraints are indispensable, as unconstrained global prompting consistently fails to localize subtle logical errors. Beyond outperforming global evaluation, our approach fundamentally alters the failure taxonomy. Error analysis reveals that, rather than exhibiting severe logical hallucinations, remaining rejections are primarily instances of "pedantic hyper-rigor" stemming from unstated domain conventions, effectively exposing implicit ambiguities within the expert benchmark itself. Our findings suggest that prompting agents to organize their verification notes in a cautious, human-mathematician-like manner can substantially improve their ability to distinguish rigorous proofs from flawed ones, with the potential to strengthen agentic reasoning on frontier mathematical concepts that the base model does not already know well, and to lay a theoretical foundation for future automated proof-review systems. Code and prompts are available at GitHub.