Decode-Time Grammars: Constrained LLM Generation over a Refinement Order of Grammar Fragments

2026-07-20

AI Summary

Large language models often generate semantically incorrect code in low-resource programming languages due to undefined variable references, invalid fields, or unsupported options. This work proposes a runtime environment-aware decoding mechanism that dynamically instantiates grammar fragments from an environment Γ, employs a region-based policy to select valid syntactic structures, and resolves open references by filling Γ-typed slots, thereby guaranteeing both syntactic well-formedness and semantic validity while enabling immediate feedback for newly declared constructs. We formalize environment-indexed grammars and their refinement order, prove their preservation of semantic correctness, and characterize the boundary of mask-enforceable properties. Experiments on TileLang, SQL, and P4 demonstrate that the gproj system eliminates phantom references with minimal overhead and substantially improves the semantic correctness of generated code.

Abstract

Large language models now write a growing share of the world's code, increasingly inside agents and serving systems that compile, execute, or dispatch generated code without line-by-line review. This works well for mainstream languages but remains brittle for low-resource programming surfaces such as domain-specific languages, custom library APIs, and command-line tools. Even under grammar-constrained decoding, a model can still produce references invalid in the current environment: a buffer never declared, a column absent from the schema, a function the library does not provide, or an unsupported CLI option. This paper introduces decode-time grammars: grammar fragments instantiated during generation from a runtime environment Gamma. A region-specific policy selects a fragment for each hole, and a tightening operator replaces open reference positions with Gamma-typed slots whose candidates are exactly the names, fields, APIs, or options available at that point. Newly generated declarations enter Gamma before later regions are decoded, so the constraining grammar can depend on the prefix already generated. This ensures not only grammatical correctness but also semantic correctness, by preventing references to undefined symbols. We formalize grammar fragments as environment-indexed grammars ordered by refinement, prove No-Ghost soundness for Gamma-slotted fragments, show that refinement preserves this support-set guarantee, and characterize the boundary of mask-enforceable properties. We implement the approach in gproj with offline grammar induction and online policy resolution. Across TileLang, SQL, and P4, with models from 0.6B to 236B parameters, gproj eliminates ghost references by construction at moderate overhead over standard constrained decoding.

Problem

• constrained decoding
• semantic correctness
• ghost references
• domain-specific languages
• runtime environment

Innovation

• decode-time grammars
• grammar-constrained decoding
• runtime environment
• semantic correctness
• ghost reference elimination