Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

2026-09-29

AI Summary

This study addresses the insufficient reliability of self-verification in terminal agents, which hinders effective error detection and correction. We construct a diagnostic framework to quantify verification behaviors, revealing that the core weakness lies in deficient error detection and repair capabilities. Accordingly, we propose Student-Conditional Verification Distillation (SCVD), a novel strategy leveraging interaction context to enhance error correction through large language model fine-tuning. Evaluated on the TerminalBench 2.1 benchmark, SCVD significantly outperforms both baseline methods and full-trajectory distillation approaches in terms of Pass@1, while demonstrating superior cross-task generalization.

Abstract

Terminal agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the agent's subsequent verification and recovery behavior. Applying it to ten terminal agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43\% of incorrect candidates are detected and only 49.36\% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves \textsc{Pass@1} on TerminalBench2.1 by 9.74--16.85 percentage points over the corresponding base models and by 4.49--8.61 points over the standard full-trajectory distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory distillation on SWE-bench Verified.

Problem

• Terminal Agents
• Self-Verification
• Error Detection
• Error Repair
• Command-line Environments

Innovation

• Self-Verification
• Diagnostic Framework
• Student-Conditioned Verification Distillation
• Knowledge Distillation
• Terminal Agents