Enhancing Automated Loop Invariant Generation for Complex Programs with Large Language Models

2024-12-13

arXiv.org

AI Summary

Traditional loop invariant generation tools exhibit limited precision and applicability on real-world programs where complex data structures intertwine with intricate control flow. Method: This paper proposes ACInv, the first static-analysis-driven, LLM-augmented framework for invariant synthesis. It extracts loop semantic features to construct structured prompts for LLM-based candidate invariant generation, and introduces an LLM-powered semantic evaluator that dynamically refines candidates via strengthening, weakening, or rejection. Contribution/Results: ACInv is the first approach to support template-level invariant generation for user-defined data structures. Evaluated on benchmarks containing complex data structures, ACInv achieves a 21% higher overall solving rate than AutoSpec, while matching its performance on purely numeric programs. Moreover, the generated invariants are reusable and significantly improve practicality for industrial-scale program verification.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageConstraint Satisfaction and Optimization: Satisfiability Modulo TheoriesMachine Learning: Large Multimodal Models (LMMs)

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Automated program verification has always been an important component of building trustworthy software. While the analysis of real-world programs remains a theoretical challenge, the automation of loop invariant analysis has effectively resolved the problem. However, real-world programs that often mix complex data structures and control flows pose challenges to traditional loop invariant generation tools. To enhance the applicability of invariant generation techniques, we proposed ACInv, an Automated Complex program loop Invariant generation tool, which combines static analysis with Large Language Models (LLMs) to generate the proper loop invariants. We utilize static analysis to extract the necessary information for each loop and embed it into prompts for the LLM to generate invariants for each loop. Subsequently, we employ an LLM-based evaluator to assess the generated invariants, refining them by either strengthening, weakening, or rejecting them based on their correctness, ultimately obtaining enhanced invariants. We conducted experiments on ACInv, which showed that ACInv outperformed previous tools on data sets with data structures, and maintained similar performance to the state-of-the-art tool AutoSpec on numerical programs without data structures. For the total data set, ACInv can solve 21% more examples than AutoSpec and can generate reference data structure templates.

Problem

• Enhancing loop invariant generation for complex programs
• Addressing challenges from mixed data structures and control flows
• Improving automated program verification with LLM integration

Innovation

• Combines static analysis with LLMs
• Uses LLM-based evaluator to refine invariants
• Generates enhanced invariants through prompt embedding