Specula: Scaling formal specifications for autonomous model checking of system code

2026-07-28

AI Summary

Traditional formal methods rely on manually crafted specifications, which do not scale to large systems, while existing large language model (LLM)-based approaches are prone to hallucination and reward hacking. This work proposes the first fully automated agent system that leverages LLMs to autonomously generate high-quality TLA+ specifications—including invariants and abstract models—and employs a self-evolutionary iterative optimization mechanism to continuously refine specification quality, enabling end-to-end automated model checking. The approach effectively mitigates LLM hallucination and substantially lowers the barrier to formal verification. Evaluated on 48 open-source projects, it uncovered 249 deep semantic bugs, most of which are elusive to current techniques, and has already been adopted in practice by multiple industrial partners.

Abstract

Specula is a push-button agentic system that generates high-quality formal specifications for large, complex system code and uses the specifications for highly effective model checking and bug finding. Specula employs large language model (LLM) based coding agents to autonomously develop TLA+ specifications, including invariants that describe correctness properties of the target system and formal models that describe the system implementation with the right level of abstractions. Specula is fully autonomous and thus eliminates the barrier of applying formal methods to real-world system code (as in traditional human-centric approaches). Meanwhile, Specula addresses limitations of LLM-driven techniques like reward hacking and hallucinations through self-evolving loops that iteratively improve specification quality by enabling the agents to deepen their understanding of system code and its behaviors. We have used Specula to check 48 open-source system projects; Specula found 249 bugs including many deep bugs that are hard to find by existing approaches. Specula has been used by several companies and is maintained at https://github.com/specula-org/Specula.

Problem

• formal specifications
• model checking
• system code
• autonomous verification
• bug finding

Innovation

• formal specification
• autonomous model checking
• large language models
• TLA+
• self-evolving agents