FVSpec: Real-World Property-Based Tests as Lean Challenges

2026-05-31

AI Summary

This work addresses the lack of effective evaluation benchmarks for AI models in real-world software formal verification. It presents the first large-scale effort to automatically translate real-world Python property-based tests (PBT) into formal specifications in Lean 4, establishing a reproducible benchmark. By integrating a multi-agent large language model pipeline with Python semantic modeling and dependent type programming techniques, the authors successfully generated 9,415 Lean 4 specifications corresponding to 2,772 tests derived from 11,039 original PBTs. The entire codebase and dataset are publicly released. This contribution substantially advances the practical application of AI-assisted formal verification in real software systems.

Abstract

We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,772 of them (25%) into 9,415 Lean 4 specifications with sorry placeholders (about 3 formalizations/PBT; we retain multiple attempts when none dominates on quality metrics). Translating PBTs into Lean specifications is challenging: it requires modeling Python semantics in Lean, inferring the logical property encoded in an imperative PBT, and handling the inherent difficulties of dependently-typed programming in a seldom-used language. We describe a three-agent LLM pipeline for transpiling PBTs into Lean specifications, evaluate coverage and quality metrics, and provide baselines for proof generation using several automated and model based approaches. All code (scraper and agents) and data (PBTs and Lean specifications) are open source. Our benchmark aims to drive progress on the underexplored problem of AI-assisted formal verification of real-world software, which is of increasing interest as AI produces more and more of the world's code.

Problem

• formal verification
• property-based testing
• AI-assisted verification
• Lean 4
• software specification

Innovation

• property-based testing
• formal verification
• Lean 4
• AI-assisted translation
• LLM pipeline