Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation

2026-07-28

AI Summary

This study investigates whether large code models genuinely leverage test cases as executable specifications or merely treat them as contextual prompts. Through behavioral and representational analyses—including hidden test pass rates, task-level behavioral flips, and layer-wise hidden state dynamics—the work systematically evaluates the impact of visible tests on model behavior. It reveals a dual mechanism by which tests operate: semantic guidance and prompt perturbation—and demonstrates that representational shift does not necessarily indicate effective test utilization. Employing linear probes, diverse test variants (assertion-based, irrelevant, synthetic), and controlled prompt lengths, experiments show that while visible tests improve performance on some tasks, the gains are inconsistent; moreover, increasing test count induces greater representational shifts without reliably enhancing accuracy and can even trigger detrimental behavioral flips.

Abstract

Public tests are widely used to guide large language model code generation, but whether models treat them as executable specifications or merely as extra prompt context remains unclear. We study test-driven code generation on HumanEval+, MBPP+, and recent LiveCodeBench tasks using Qwen2.5-Coder-7B and Qwen3.6-27B. We compare natural-language-only prompts with relevant visible tests, shuffled outputs, irrelevant tests, assertion-only tests, and stronger-model-generated synthetic tests. Evaluation combines hidden or private test pass rates with task-level behavior flips, linear probes, and layer-wise hidden-state shifts. Visible tests substantially improve Qwen2.5 performance on MBPP+ but have little or unstable effect on HumanEval+ and LiveCodeBench. For Qwen3.6 on LiveCodeBench, the natural-language-only pass rate rises from 13.1% to 39.4%, yet relevant original tests add only 2.9 percentage points (p = .458), and synthetic high5 tests add 1.7 points over their matched baseline (p = .701). More tests produce larger representation shifts, but accuracy gains quickly saturate or disappear; assertion-only and irrelevant-test prompts can strongly alter hidden states without improving correctness. After controlling for prompt length, task-level shifts are associated with both beneficial and harmful flips, with a substantially stronger association for harms. These results show that tests influence code models through both semantic guidance and prompt-context perturbation, and that representational change alone does not demonstrate effective test utilization.

Problem

• code language models
• test-driven code generation
• executable specifications
• prompt context
• model behavior

Innovation

• test-driven code generation
• code language models
• behavioral analysis
• representational probing
• prompt perturbation