Sharding Prevents LLM Oversight Failures and Adversarial Exploitation

2026-08-05

AI Summary

This work addresses the challenge that large language models often exhibit reduced alignment with expert judgments and heightened susceptibility to adversarial presentation formats when performing multi-task reasoning within a single inference call, primarily due to insufficient evidentiary grounding. To mitigate this, the authors propose a “judgment sharding” mechanism that decomposes complex tasks into constituent subtasks, each processed by independent model invocations whose outputs are subsequently aggregated. Crucially, this approach operates within the same total computational budget yet significantly enhances both supervised accuracy and robustness. The method enables weaker models to outperform stronger monolithic baselines and effectively alleviates judgment overload–induced adversarial vulnerabilities. Empirical evaluations in domains such as legal reasoning and clinical trial assessment demonstrate substantial improvements in expert consistency, suppression of adversarial over-acceptance, and sustained low error rates even under attacks that expand the search space.

Abstract

Giving an LLM judge more compute does not necessarily make it check more requirements. When one call must return many verdicts, some decisions become weakly grounded in the evidence, even when that call receives the same token or tool budget as a panel of separate calls. Across expert-graded research replications, legal work, and clinical-trial assessments, agreement with experts falls as the number of verdicts per call grows. We identify sharding as the intervention that mitigates this failure in model-based oversight. Sharding partitions the requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts. Against a single call with the panel's full budget, sharding improves agreement while holding the model, evidence, total budget, and per-decision budget fixed. Overall, we find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget. Additionally, we find that sharding exhibits robustness against adversaries. A best-of-N adversary can hold the underlying work fixed, vary only its presentation, and increase an overloaded judge's acceptance of genuinely unmet criteria severalfold. Wherever sharding reduces baseline error, it removes this adversarial advantage, keeping over-acceptance low even as the adversary's search widens. Sharding does not address attacks that persuade the judge separately on each criterion rather than exploiting overload. In that setting, we find that debate-style opposition on top of sharding withstands such adaptive re-optimization.

Problem

• LLM oversight
• sharding
• adversarial exploitation
• judgment overload
• model reliability

Innovation

• sharding
• LLM oversight
• adversarial robustness
• model-based evaluation
• decision partitioning