Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges

2026-06-03

AI Summary

This study addresses the vulnerability of large language models (LLMs) acting as evaluators to post-decision manipulation through interactive challenges, which can compromise judgment reliability. Introducing the novel concept of “post-decision manipulability,” the work proposes an Evaluative Robustness Score (ERS) to quantify LLM stability and directional susceptibility under adversarial interaction. Through controlled experiments, counter-baseline challenge protocols, target-verification balanced designs, and authoritative framework testing—systematically evaluated on MT-Bench and AlpacaEval benchmarks—the findings reveal that while LLM judges remain stable under neutral re-evaluation, their judgments undergo significant reversal when subjected to targeted challenges. This leads to reduced alignment with human preferences, shifts in model rankings, and even high-confidence harmful misjudgments, underscoring critical robustness limitations in current LLM-based evaluation paradigms.

Abstract

LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipelines typically assume that judgments are stable properties of fixed inputs. We show that this assumption does not hold under interaction. We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge after an initial decision has been made. Across controlled experiments on MT-Bench and AlpacaEval, we find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge. An anti-baseline challenge protocol shows that stable judgments can be overturned through motivated interaction, while a counterbalanced target-validation protocol separates this reversibility from net target-directed steering. These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence. Authority framing is especially destabilizing, and revised judgments are often accompanied by low-overlap justifications, suggesting post hoc rationalization rather than reliable error correction. We introduce the Evaluation Robustness Score (ERS) to quantify interactional robustness by combining reversal susceptibility with counterbalanced directional effects. Our findings identify post-decision interaction as a distinct failure mode for LLM-as-judge evaluation and motivate evaluation protocols that measure not only static agreement, but robustness under challenge.

Problem

• post-decision manipulability
• LLM-as-judge
• evaluation robustness
• judgment stability
• interactive evaluation

Innovation

• post-decision manipulability
• evaluation robustness
• LLM-as-judge
• interactional stability
• Evaluation Robustness Score