Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference

2025-02-22

AI Summary

This work investigates the capability of large language models (LLMs) in program semantic understanding and formal specification synthesis for correctness verification. To this end, we introduce FormalBench—the first comprehensive benchmark tailored to formal specification inference—covering core challenges such as loop reasoning and semantics-preserving transformations. Experimental results show that while LLMs perform well on simple control-flow structures, their robustness degrades significantly on complex loops and semantic equivalence transformations. Building on these findings, we propose a self-healing prompting strategy that iteratively validates and refines generated specifications, improving synthesis success rate by 25%. Our study provides the first systematic characterization of LLMs’ capabilities and limitations in program semantic reasoning. Moreover, it delivers a reproducible evaluation framework and effective intervention techniques to advance LLMs’ formal reasoning capacity.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageMachine Learning: Large Multimodal Models (LMMs)Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Large language models for searchUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly being used to automate programming tasks. Yet, LLMs' capabilities in reasoning about program semantics are still inadequately studied, leaving significant potential for further exploration. This paper introduces FormalBench, a comprehensive benchmark designed to evaluate LLMs' reasoning abilities on program semantics, particularly via the task of synthesizing formal program specifications to assist verifying program correctness. This task requires both comprehensive reasoning over all possible program executions and the generation of precise, syntactically correct expressions that adhere to formal syntax and semantics. Using this benchmark, we evaluated the ability of LLMs in synthesizing consistent and complete specifications. Our findings show that LLMs perform well with simple control flows but struggle with more complex structures, especially loops, even with advanced prompting. Additionally, LLMs exhibit limited robustness against semantic-preserving transformations. We also highlight common failure patterns and design self-repair prompts, improving success rates by 25%.

Problem

• Evaluate LLMs' ability to reason about program semantics.
• Assess LLMs' performance in synthesizing formal program specifications.
• Identify LLMs' limitations with complex program structures and robustness.

Innovation

• FormalBench evaluates LLMs on program semantics.
• LLMs synthesize formal specifications for program verification.
• Self-repair prompts improve specification success rates.