Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents

2026-09-25

AI Summary

This study addresses the challenges of sparse terminal rewards, high inference costs, and training instability in long-horizon coding agents by proposing Contextual Early Reward (CER). This method predicts terminal rewards based on behavioral evidence from trajectory prefixes and introduces a novel approach that synthesizes adaptive scoring criteria from historical experience. By enabling accurate outcome prediction without full execution, CER facilitates efficient dense evaluation. Experimental results demonstrate that CER improves RM@8 by 4.2 percentage points on SWE-bench while reducing token consumption by 52.7%, significantly enhancing both training efficiency and model performance.

Abstract

Long-horizon coding agents receive verifiable rewards only after completing expensive sequences of tool calls. This increases inference cost, amplifies early wrong hypotheses, and can lead to sparse terminal reward and unstable training. We introduce Contextual Early Reward (CER), which predicts terminal reward through behavioral evidence in a trajectory prefix. CER synthesizes adaptive rubrics specific to the current task and stage through experiences summarized from related historical tasks. In test-time scaling on SWE-bench Verified, CER improves RM@8 over the strongest baseline by 4.2 percentage points (pp) on Nemotron 3 Ultra and 2.0 pp on Qwen 3.6 27B; on Nemotron, it takes only 15.3% tokens to match the best baseline performance. In RL training experiments, CER exceeds full-rollout TMax by 1.9 pp while using 52.7% fewer online policy-and-judge tokens. Together, CER provides an interpretable, efficient, and dense evaluation method for long-horizon coding agents.

Problem

• Long-horizon coding agents
• Sparse terminal reward
• Inference cost
• Training instability
• Reward prediction

Innovation

• Early Terminal Reward Prediction
• Contextual Early Reward (CER)
• Long-horizon Coding Agents
• Test-time Scaling
• Adaptive Rubrics