The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards

2026-07-27

AI Summary

Current static leaderboards evaluate only factual accuracy while ignoring computational cost, making it difficult to distinguish between efficient systems and those with high overhead. This work proposes the MAS-HQ protocol, which for the first time integrates computational overhead into the factual evaluation framework. By employing a resource-aware multi-agent adversarial mechanism—incorporating normalized cost modeling, adversarial matching, and sensitivity analysis—it establishes a reproducible, holistic evaluation pipeline. Evaluated on summarization and open-domain question answering tasks, the approach significantly outperforms single-agent baselines, consistently elicits efficient strategies, and maintains strong discriminative power among state-of-the-art models. The unified Q-Score metric jointly quantifies both factual accuracy and efficiency.

Abstract

On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource-aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5 in simulated preview) whose raw factuality scores are already bunched near the ceiling. MAS-HQ provides a reproducible way to measure how much a factual answer costs.

Problem

• hallucination
• factuality
• resource-aware evaluation
• compute cost
• leaderboard

Innovation

• resource-aware evaluation
• hallucination benchmarking
• Q-Score
• multi-agent competition
• cost-normalized factuality