structured output

This work addresses the low compliance and lack of systematic evaluation of constrained decoding techniques under realistic, complex constraints—particularly JSON Schema. We introduce JSONSchemaBench, the first large-scale structured generation benchmark comprising 10K real-world JSON schemas, and propose a multidimensional evaluation framework assessing compliance, constraint coverage, and output quality across six state-of-the-art methods. Our analysis reveals, for the first time, that compliance rates drop by over 40% for existing frameworks when handling nested, recursive, and conditional constraints. We further identify XGrammar and Outlines as achieving the best trade-off between inference efficiency and generation quality. The benchmark and evaluation suite are fully open-sourced, filling a critical gap in systematic evaluation for structured generation and advancing constrained decoding toward higher reliability and stronger generalization.

This study addresses the frequent failure of large language models (LLMs) in software engineering tasks due to structurally invalid outputs—such as syntactic or formatting errors—that prevent correct parsing by downstream toolchains, even when the semantic content is accurate. The authors systematically evaluate the reliability of structured output generation across four representative tasks, categorizing errors into syntactic, structural, and semantic types. They propose a template-driven token-matching generation (TTMG) method that enforces structural consistency during autoregressive decoding. Experimental results demonstrate that TTMG nearly eliminates syntactic errors; however, structural and semantic errors remain prevalent. This work reveals, for the first time, that the fundamental bottleneck in structured output generation lies not merely in syntax but in the insufficient coordination between structural and semantic correctness, indicating that existing structural control mechanisms, while necessary, are insufficient without joint guarantees of both dimensions.

Large language models (LLMs) exhibit limited capability in generating structured outputs strictly compliant with JSON Schema, primarily due to bottlenecks in schema understanding, string escaping handling, and natural-language-to-schema mapping. Method: We introduce SchemaBench—the first large-scale, high-coverage JSON Schema benchmark comprising over 40,000 diverse schemas—and propose a schema-aware reinforcement learning framework guided by a fine-grained syntactic validator, integrated with structured prompting for end-to-end optimization. Contribution/Results: Our approach significantly improves both JSON syntactic validity and schema adherence rates, substantially outperforming state-of-the-art baselines on SchemaBench. Moreover, it delivers measurable gains in downstream practical applications, such as API call generation and execution, demonstrating robust generalization across schema complexity and domain diversity.

To address the error-prone and inefficient manual rewriting of data transformation logic upon JSON Schema evolution, this paper proposes a type-directed, top-down program synthesis approach for automatically generating semantics-preserving JSON Schema converters. Our method integrates type inference, semantic constraint modeling, a rewrite system, and intermediate representation (IR)-driven code generation to guarantee lossless data transformation and formal verifiability. It natively supports complex nested schemas and synthesizes correct, efficient, and human-readable Python and JavaScript conversion code. We evaluate our approach on real-world API configuration schemas and healthcare data integration scenarios, demonstrating its safety—via formal guarantees and empirical validation—its practical utility in industrial settings, and its generalizability across diverse schema evolution patterns. Experimental results confirm high accuracy, robustness to structural changes (e.g., field additions, type refinements, nested object restructuring), and scalability to large, deeply nested schemas.

Large language models (LLMs) exhibit insufficient reasoning capabilities for complex programming tasks: process supervision relies on costly and error-prone reward modeling, while outcome supervision struggles to coordinate multi-step reasoning. To address this, we propose a novel “outcome-refinement-as-process” supervision paradigm that eliminates explicit reward modeling and instead leverages program execution feedback—such as runtime outputs and error traces—as label-free, reliable intermediate supervision signals. Our approach integrates tree-based multi-path exploration with a lightweight model adaptation framework to enable efficient, execution-guided reasoning. Evaluated across five LLMs and three benchmark datasets, our method achieves average improvements of 26.9% in code correctness and 42.2% in execution efficiency. Notably, it significantly boosts the performance of smaller models on algorithmic competition–style tasks. This work establishes a scalable, low-overhead paradigm for complex programming reasoning, grounded in direct execution feedback rather than surrogate reward signals.

This study addresses the overlooked trade-off between output validity and factual correctness when small language models are subjected to hard structural constraints—such as JSON formatting or tool-call schemas—which significantly degrade answer accuracy. The authors introduce "constraint tax," a metric protocol that quantifies the performance penalty imposed by such constraints under fixed model, task, and input conditions, and propose a novel delayed-enforcement paradigm: first allowing unconstrained reasoning, then applying structural constraints post-hoc. Experiments on Qwen2.5 and SmolLM2 models reveal that while hard constraints achieve 100% format compliance, they reduce answer accuracy from 19.7% to 11.0%, with 88.9% of outputs being syntactically valid yet factually incorrect. In calendar-based tool-calling tasks, executable accuracy plummets from 91.5% to 48.0%.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.

This work addresses the frequent lack of semantic validity in code generated by large language models (LLMs) for software engineering tasks. To this end, it introduces a projection decoding framework that, for the first time, treats graph-based representations as first-class citizens alongside textual sequences during generation. The framework incrementally constructs partial graph structures in parallel with token prediction, directly embedding domain-specific semantics into the decoding process. This integration enables uncertainty modeling, incremental semantic validation, and provable correctness guarantees, thereby establishing a verifiable foundation for LLM-driven software engineering automation. Experimental results demonstrate that the proposed approach significantly improves the semantic validity of generated artifacts in program synthesis tasks.

This work addresses the challenge of reliably translating natural language into industrial-grade, deployable SysMLv2 models. The authors propose an iterative generate-check-repair framework that, for the first time, integrates a production-level SysMLv2 conformance checker directly into the generation process as a control mechanism rather than a post-processing step. By combining large language model (LLM) generation with deterministic diagnostic feedback and targeted repair strategies—and terminating only when zero errors remain—the method achieves perfect compliance. Evaluated across 604 test cases derived from 151 prompts and four distinct LLMs, the approach elevates single-pass generation compliance from 51.16% to 100%, enabling robust, direct translation of natural language specifications into engineering-ready SysMLv2 models.