MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc

2026-08-04

AI Summary

This work addresses critical flaws in the original long-document question answering benchmark MMLongBench-Doc, which suffers from erroneous, ambiguous, or incomplete annotations and relies on string-matching–based evaluation metrics that fail to capture semantic equivalence, leading to distorted assessments. To remedy these issues, the authors manually correct annotations for 106 questions, remove 11 invalid samples, and introduce a large language model–based semantic equivalence judgment mechanism to replace exact string matching. They further design a reusable empty-set key expansion protocol and a transparent annotation revision logging system that clearly distinguishes negative samples from expandable entries. The resulting revised benchmark, MMLongBench-Doc-V2, comprises 1,071 questions across 134 documents and is accompanied by comprehensive correction logs and an evaluation toolchain, substantially enhancing assessment accuracy, semantic fidelity, and reproducibility.

Abstract

MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs. Two properties of it push measured scores away from the quantity they are meant to capture: the reference metric compares extracted answers, so 1,358,000 loses to 1358000; and a non-trivial share of ground-truth annotations are wrong, ambiguous, or incomplete --- concentrated, because of how they were found, in exactly the questions capable systems answer correctly. MMLongBench-Doc-V2 corrects 106 annotations, each published with the page and arithmetic that settle it, and replaces the string metric with a pinned LLM judge asked whether a response means the reference. Ten questions whose document ships under the wrong filename are removed rather than counted wrong, along with one duplicated question, leaving 1,071 questions over 134 documents. The most reusable contribution is a decision procedure for when an empty set key may be widened and when widening would destroy a deliberate negative sample; applied to all 208 rows, it widened 14. V2 scores are not comparable with published V1 numbers. The corrected corpus, the per-entry correction record and the evaluation harness are available at https://github.com/VectifyAI/MMLongBench-Doc-V2.

Problem

• long-document QA
• annotation errors
• evaluation metric
• semantic equivalence
• benchmark correction

Innovation

• semantic-aware evaluation
• annotation correction
• LLM-based judging
• negative sample preservation
• long-document QA benchmark