SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

2026-07-21

AI Summary

This work proposes the first cost-aware routing framework for supervised fine-tuning data acquisition that integrates statistical gating with an adversarial adjudication mechanism to efficiently identify high-value corpora while avoiding costly misacquisitions. The approach evaluates candidate samples along three axes—diversity, utility, and redundancy—using low-cost statistical estimates for initial filtering and triggering a multi-agent debate between proponent and opponent advocates only when confidence is insufficient. Evaluated through quality assessments with confidence intervals and controlled synthetic benchmarks, the system achieves 0.90 accuracy and 0.83 F₁ score across twelve datasets at a unit cost of just $0.017, substantially outperforming always-verify strategies. Moreover, it provides the first quantitative evidence of stance bias (52% stance reversal) and oppositional advantage (80% win rate) in LLM-based adjudication.

Abstract

Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present \sys{}, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes -- diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets ($2{\times}3{\times}2$ grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 $F_1$ at \$0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate (\$0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 ($p\approx3{\times}10^{-6}$) and a 52\% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work.

Problem

• supervised fine-tuning
• data procurement
• quality assessment
• cost-aware decision
• trustworthy AI

Innovation

• statistics-first gating
• adjudicative escalation
• SFT data procurement
• cost-aware routing
• measurement fidelity