Efficient Grammar-Constrained Decoding via Parser Stack Classification

2026-08-03

AI Summary

Existing grammar-constrained decoding methods suffer from high computational overhead under large vocabularies, severely limiting the throughput efficiency of large language models in generating structured text. This work proposes PSC (Parsing Stack Classifier), the first approach to achieve vocabulary-size-independent mask computation complexity. By preprocessing all token-level syntactic acceptance conditions into a unified parsing stack classifier, PSC enables the generation of a complete constraint mask at each decoding step through a single stack state check. Integrating context-free grammars, parsing stack modeling, and an efficient classifier design, the method achieves speedups of up to 700× and 30× on programming language and JSON generation tasks, respectively, with end-to-end throughput approaching that of unconstrained decoding.

Abstract

LLMs are widely used to generate structured output like source code or JSON. Grammar-constrained decoding (GCD) can guarantee the syntactic validity of the generated output, by masking out tokens that violate rules specified by a context-free grammar. However, the online computational overhead of existing GCD methods, with latency typically scaling linearly with vocabulary size, limits the throughput of LLMs, especially for models with large vocabularies. To address this issue, we propose PSC, a novel grammar-constrained decoding method. By combining acceptance conditions of all vocabulary tokens into a single classifier of the parser stack during preprocessing, PSC can compute the complete vocabulary mask by checking the parser stack exactly once per decoding step, with time complexity independent of the vocabulary size. Experiments show that PSC computes masks up to 700$\times$ faster than baselines on complex programming language grammars, and up to 30$\times$ faster for schema-conformant JSON; end-to-end LLM throughput with PSC approaches that of unconstrained decoding. We analyze the preprocessing overhead for preprocessing providers and decoding users, and provide a break-even point analysis to help users decide whether to do preprocessing by themselves.

Problem

• grammar-constrained decoding
• large language models
• structured output
• computational overhead
• vocabulary size

Innovation

• grammar-constrained decoding
• parser stack classification
• vocabulary masking
• efficient decoding
• structured output generation