🤖 AI Summary
This study investigates whether large language models can generate executable formal specifications in JavaScript that faithfully capture the behavior of real-world systems, and systematically evaluates the impact of programming language choice, specification structure, and prompting strategies on fidelity. To this end, we introduce JS-SAM—the first executable JavaScript specification backend based on the SAM pattern—and conduct controlled experiments on the SysMoBench benchmark to disentangle the effects of language, contract, and prompting. Our findings reveal that specification contracts, rather than the host language itself, primarily govern fidelity; JS-SAM-generated specifications, when semantically mapped to TLA+, achieve fidelity comparable to native TLA+ specifications; only system consistency validation effectively discriminates model performance; and semantic understanding remains the principal bottleneck for complex protocols such as consensus algorithms. This work establishes a complementary verification paradigm pairing JS-SAM with TLA+.
📝 Abstract
Can large language models write faithful formal specifications of real systems, and does it matter whether they write in a formal language they have seen rarely or in a mainstream language abundant in their training data? We study this on SysMoBench, which grades a generated specification in four phases, the decisive one replaying execution traces captured from the running system. We add JS-SAM, its first non-formal backend, in which a specification is executable JavaScript written in the SAM pattern, a pattern whose semantics mirror TLA+, and run a controlled comparison that separates three variables an ordinary head-to-head entangles: the language, the specification contract (the shape the model must fill), and the prompt. The study spans four frontier models and three systems (an operating-system spinlock, a distributed lock service, and the Etcd Raft consensus implementation), with counterexample-driven repair. Three findings emerge. First, conformance against the real system is the only phase that discriminates among models; internal consistency is inexpensive to satisfy, and a specification that looks right is not thereby right. Second, once the comparison is drawn like for like, the specification contract, not the language, governs fidelity: JavaScript in the shape of the TLA+ transition relation is as faithful as TLA+. Third, a minimal contract carries transcription but not semantic derivation: at consensus scale the difficulty becomes understanding the protocol, which no contract shape and no language supplies. We frame executable JavaScript as a checkable specification substrate that complements, rather than replaces, the verification TLA+ provides, and present the study as a case study.