TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

2026-10-07

AI Summary

This study addresses the issues of reward hacking and evaluation distortion caused by fixed unit tests in LLM code benchmarks by proposing a dynamic auditing framework. Overcoming the limitations of static testing, this framework introduces a trial-feedback-driven evaluator evolution mechanism. By integrating agent-based code generation, dynamic test synthesis, and lightweight sampling audit techniques, it adaptively generates supplementary tests for each trial to verify patch compliance while supporting reproducible failure validation. Experimental results across multiple frontier models and benchmarks reveal that 34.4% of originally correct judgments were actually non-compliant, reducing the overall resolution rate to 33.2%. These findings expose significant deficiencies in existing evaluation paradigms.

Abstract

Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and reuses the resulting tests across all trials for the same task. Across 6 frontier model backends and 5 benchmarks such as DeepSWE and SWE Marathon, we find that about 34.4% of the model trials currently judged correct violate the task requirements, lowering the overall resolution rate from 50.6% to 33.2%. Our results reveal a fundamental limitation of current coding-agent evaluation: as LLMs become better at optimizing against fixed evaluators, those evaluators themselves must become more adaptive.

Problem

• coding benchmarks
• LLM agents
• evaluation reliability
• reward hacking
• unit tests

Innovation

• Agentic Coding Benchmarks
• Evaluator Evolution
• Dynamic Test Generation
• Reward Hacking
• TestJack