Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation

2026-10-06

AI Summary

This study addresses the lack of deterministic verification of user intent in formal specification generation, which risks producing proofs grounded in erroneous specifications. To this end, it constructs a unified dataset and multidimensional evaluation framework encompassing formal validity, similarity, and behavioral adequacy. The work proposes a strategy distinguishing input acceptance from output constraints to clarify the evidential scope of metrics, and integrates an LLM agent workflow, the Lean theorem prover, and Generalized Tree Edit Distance (GTED) to enable automated evaluation. The findings reveal the limitations of single similarity metrics and the impact of metric coverage on ranking outcomes, demonstrating that perfect postcondition scores may obscure deficiencies in input contracts. Ultimately, this research provides a systematic benchmark for evaluating formal specifications.

Abstract

When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent. A checked proof can therefore establish correctness against a specification that misrepresents the intended behaviour. We take a step towards holistic SpecGen evaluation with a unified dataset assembled from $350$ existing Lean tasks, including $189$ from VERINA and $161$ from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy. We distinguish acceptance of required inputs from acceptance of valid outputs and rejection of invalid outputs, while making each metric's evidence scope explicit. Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to $32$ jointly measurable VERINA tasks changes the VERINA configuration's position from second to fourth in mean similarity, showing the importance of measurement coverage. In an authored control, a specification achieves $100\%$ positive test recall and negative test rejection while accepting $0\%$ of required inputs. This demonstrates that perfect postcondition scores can miss an unusable input contract, motivating separate feedback on input coverage and output constraints.

Problem

• Formal Specification Generation
• SpecGen Evaluation
• Verifiable Code
• Intent Alignment

Innovation

• Specification Generation
• Holistic Evaluation
• Formal Verification
• Generalized Tree Edit Distance
• LLM