Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

2026-07-21

AI Summary

This work addresses a critical gap in current evaluation methodologies for generated content: while existing approaches focus on factual accuracy, they struggle to assess factual completeness—whether a response encompasses all necessary information. To bridge this gap, the authors propose a two-tiered meta-scoring framework that structurally models the hierarchical organization and relative importance of required answer components, enabling the automatic generation of binary checklists reliably evaluable by large language models. They introduce GAMUT, the first benchmark specifically designed for evaluating factual completeness, comprising 1,813 questions derived from wearable device images alongside expert-validated ground-truth criteria. Experiments across 14 state-of-the-art models demonstrate that GAMUT is highly challenging (with a top score of 58.7%), exhibits strong discriminative power, and remains robust to the choice of scoring model, thereby overcoming the limitations of traditional flat Boolean verification schemes.

Abstract

Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: it requires enumerating the full set of facts a complete answer should contain, and these facts rarely form a flat list. They often involve open-ended sets where coverage is what matters, ordered processes, and relationships among facts that a list of independent boolean checks fails to capture. We introduce a two-level meta-rubric framework for evaluating open-ended generation, and instantiate it as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. The framework rests on a two-level rubric representation: a structured meta-rubric captures the organization and importance of the required content, which is then mechanically compiled into a flat checklist of binary, machine-gradable rubrics that an LLM judge scores reliably. We construct 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Because the framework is modality-agnostic, we also release a text-only variant. Evaluating 14 frontier and open-weight models, we find the benchmark genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.

Problem

• factual completeness
• open-ended generation
• factuality evaluation
• long-form generation
• meta-rubric

Innovation

• factual completeness
• two-level meta-rubric
• open-ended generation
• GAMUT benchmark
• structured evaluation