Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

2026-08-05

AI Summary

This study investigates whether prematurely fixing intermediate evidence in large language model (LLM) judge evaluations compromises judgment consistency and alignment with human preferences. To address this, the authors propose a multi-stage prompting strategy that decouples evidence extraction from final judgment, implementing two-stage and three-point evidence locking on Claude Sonnet 4.5 and GPT-5. Systematic evaluation across the HelpSteer3, FeedbackQA, and CoVal datasets reveals—for the first time—that freezing intermediate evidence reduces alignment with human preferences by 4–6 percentage points and increases answer-order inconsistency by 8–10 percentage points. These findings indicate that intermediate outputs cannot reliably substitute for raw inputs in final decision-making, thereby challenging the validity of current evidence-locking mechanisms in LLM-based judging pipelines.

Abstract

LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.

Problem

• LLM-as-Judge
• evidence locking
• evaluation reliability
• human preference alignment
• judgment consistency

Innovation

• Evidence Locking
• LLM-as-Judge
• structured elicitation
• auditability
• preference alignment