Learning Context-Free Grammars for Grammar-Constrained Decoding via Declarative Agentic Programming with Guarantees

2026-08-05

AI Summary

This work addresses the challenge that large language models often generate code violating the syntax of domain-specific languages (DSLs) when invoking external services, a problem exacerbated by the absence of context-free grammars for third-party DSLs needed for syntax-constrained decoding. To overcome this, the authors propose Autogrammar, an agent that uniquely integrates Kripke structures with language models, enabling declarative control of agent behavior via linear temporal logic and automatically inducing DSL grammars from documentation and execution feedback—eliminating the need for manual grammar engineering. Evaluated on three real-world DSLs, the learned grammars achieve near-perfect precision (≈100%) on unseen data, significantly outperforming baseline methods in end-to-end task accuracy, matching or exceeding human-crafted grammars in 80% of tasks, while accelerating inference by 3.8×.

Abstract

Language models (LMs) are increasingly used to interact with external services via programs written in domain-specific languages (DSLs). Unfortunately, since DSLs are often low-resource and esoteric, LMs frequently produce syntactically invalid programs in these languages. Grammar-constrained decoding can eliminate such failures, but requires syntactic constraints. These are usually in the form of a context-free grammar for the target language, an artifact that is hard to come by for third-party DSLs. In this work, we define an agent, called Autogrammar, that automatically learns context-free grammars from documentation and execution data. Autogrammar is formalized as a Kripke structure whose nondeterministic choices are resolved by a language model, enabling declarative control of agent behavior via linear temporal logic constraints. We evaluate four versions of Autogrammar on three DSLs (i.e., Amazon CloudWatch Logs Insights, Dynatrace Query Language, and Datadog Search Syntax) and find that it generates grammars that achieve near perfect precision on unseen data; that temporal restrictions reduce execution time by 3.8x without incurring statistically-significant loss in precision; that execution data is crucial while documentation is dispensable; and that grammar-constrained decoding using Autogrammar-generated grammars significantly improves end-to-end LM performance on eight out of ten real tasks, matching or exceeding the performance of a professionally-maintained grammar. In comparison, the context-free grammars generated by existing LM baselines and a state-of-the-art formal technique perform significantly worse over the same evaluation.

Problem

• context-free grammar
• grammar-constrained decoding
• domain-specific language
• language models
• syntactic validity

Innovation

• grammar-constrained decoding
• context-free grammar induction
• declarative agentic programming
• Kripke structure
• linear temporal logic