The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

2026-04-28

AI Summary

Existing benchmarks inadequately assess the quality of structured outputs from large language models in multi-source scenarios, as they focus either on structural compliance or value correctness within a single modality. This work proposes the first cross-modal, source-agnostic evaluation framework for structured generation, which uniformly converts inputs from text, images (OCR-processed PDFs), and audio (AMI meeting transcripts) into textual contexts, constrains model outputs via JSON Schema, and constructs a complex, realistic dataset through multi-hop question answering. Evaluation of 21 state-of-the-art models across seven metrics reveals near-perfect structural compliance but markedly lower value accuracy—83.0%, 67.2%, and 23.7% for text, image, and audio sources, respectively—highlighting that extracting structured information from long, multi-source contexts remains a significant challenge.

Abstract

Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.

Problem

• structured output
• benchmark
• large language models
• multi-source evaluation
• schema compliance

Innovation

• structured output
• multi-source benchmark
• schema compliance
• value accuracy
• modality-agnostic evaluation