Towards Verified Code Reasoning by LLMs

2025-09-30

AI Summary

Current large language models (LLMs) exhibit insufficient accuracy in code reasoning tasks, limiting their deployment in high-assurance applications such as code understanding, review, and generation. To address this, we propose a novel methodology integrating natural language inference with formal verification: LLM-generated reasoning chains are automatically translated into verifiable logical representations and rigorously validated end-to-end using program analysis and formal verification tools—requiring no human intervention to assess reasoning correctness. Evaluated on 40 real-world code reasoning cases—20 uninitialized-variable detection and 20 program equivalence verification tasks—the system confirms correctness for 13 of 20 positive instances and precisely identifies defects in 6 of 8 erroneous reasonings. This substantially enhances the trustworthiness and practical utility of LLM outputs in safety-critical software engineering contexts.

Technology Category

Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While LLM-based agents are able to tackle a wide variety of code reasoning questions, the answers are not always correct. This prevents the agent from being useful in situations where high precision is desired: (1) helping a software engineer understand a new code base, (2) helping a software engineer during code review sessions, and (3) ensuring that the code generated by an automated code generation system meets certain requirements (e.g. fixes a bug, improves readability, implements a feature). As a result of this lack of trustworthiness, the agent's answers need to be manually verified before they can be trusted. Manually confirming responses from a code reasoning agent requires human effort and can result in slower developer productivity, which weakens the assistance benefits of the agent. In this paper, we describe a method to automatically validate the answers provided by a code reasoning agent by verifying its reasoning steps. At a very high level, the method consists of extracting a formal representation of the agent's response and, subsequently, using formal verification and program analysis tools to verify the agent's reasoning steps. We applied this approach to a benchmark set of 20 uninitialized variable errors detected by sanitizers and 20 program equivalence queries. For the uninitialized variable errors, the formal verification step was able to validate the agent's reasoning on 13/20 examples, and for the program equivalence queries, the formal verification step successfully caught 6/8 incorrect judgments made by the agent.

Problem

• Verifying LLM code reasoning correctness for high precision
• Automating validation of agent responses through formal verification
• Improving trust in code analysis without manual confirmation

Innovation

• Extracts formal representation from agent responses
• Uses formal verification to validate reasoning steps
• Applies program analysis tools for code verification