Formalizing Complex Mathematical Statements with LLMs: A Study on Mathematical Definitions

2025-02-17

AI Summary

This work investigates the capability of large language models (LLMs) to automatically formalize real-world mathematical definitions—sourced from Wikipedia and arXiv—into Isabelle/HOL. To address the limitation of existing benchmarks (e.g., miniF2F) in capturing definition-specific challenges, we introduce Def_Wiki/Def_ArXiv, the first dual-source benchmark for mathematical definition formalization. We propose definition grounding—a method that integrates context-aware prompting with proof-assistant feedback-driven structured refinement—and close the loop via external verification. Experiments demonstrate a 16% improvement in model self-correction ability and a 43% reduction in undefined-symbol errors. Our findings reveal that formalizing mathematical definitions is substantially more challenging than theorem proving, exposing critical bottlenecks in LLMs’ semantic precision and symbolic consistency.

Technology Category

Knowledge Representation and Reasoning: Automated Reasoning and Theorem ProvingNatural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Large language models for search

Abstract

Thanks to their linguistic capabilities, LLMs offer an opportunity to bridge the gap between informal mathematics and formal languages through autoformalization. However, it is still unclear how well LLMs generalize to sophisticated and naturally occurring mathematical statements. To address this gap, we investigate the task of autoformalizing real-world mathematical definitions -- a critical component of mathematical discourse. Specifically, we introduce two novel resources for autoformalisation, collecting definitions from Wikipedia (Def_Wiki) and arXiv papers (Def_ArXiv). We then systematically evaluate a range of LLMs, analyzing their ability to formalize definitions into Isabelle/HOL. Furthermore, we investigate strategies to enhance LLMs' performance including refinement through external feedback from Proof Assistants, and formal definition grounding, where we guide LLMs through relevant contextual elements from formal mathematical libraries. Our findings reveal that definitions present a greater challenge compared to existing benchmarks, such as miniF2F. In particular, we found that LLMs still struggle with self-correction, and aligning with relevant mathematical libraries. At the same time, structured refinement methods and definition grounding strategies yield notable improvements of up to 16% on self-correction capabilities and 43% on the reduction of undefined errors, highlighting promising directions for enhancing LLM-based autoformalization in real-world scenarios.

Problem

• LLMs' ability to formalize complex math
• Autoformalization of real-world math definitions
• Enhancing LLMs with structured refinement methods

Innovation

• LLMs for autoformalizing mathematical definitions
• External feedback enhances LLM performance
• Formal definition grounding reduces undefined errors