What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

2026-09-26

AI Summary

This study addresses the incomparability and unfairness in long-horizon agent leaderboards caused by imbalanced inputs, evidence, or budgets. It proposes a combinatorial controllability framework that employs comparison windows to identify sources of discrepancy and establishes admissibility tests to reject invalid comparisons. By leveraging out-of-window distractors to derive theoretical bounds on score differences, the framework enables fairness pre-screening. The evaluation pipeline is further optimized through the BioLitBench benchmark, statistical calibration, and reinforcement learning with stage-wise rewards. Empirically, the approach successfully rejects 11 of 21 unfair comparisons and corrects seven erroneous conclusions. Notably, the proposed SCRIBE model achieves a certified ranking interval of [1, 2] on Qwen3.8, significantly outperforming existing pipelines.

Abstract

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It refuses 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these refused pairs. The same comparison windows support stage-level training. We train SCRIBE on Qwen3.8-27B using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. Under same pool, SCRIBE matches the strongest published retriever and is certified above three published pipelines.

Problem

• leaderboard fairness
• compositional controllability
• biomedical literature-review agents
• fair evaluation
• long-horizon agents

Innovation

• Compositional Controllability
• Fair Evaluation
• Biomedical Literature-Review Agents
• Stage-level Training
• BioLitBench