verifier design

Current large language model agents struggle to identify procedural errors—such as selecting incorrect skills, skipping critical steps, or miscombining tools—when relying solely on final outcome validation. To address this limitation, this work proposes SkillCoach, a novel framework featuring a self-evolving process-scoring mechanism. It automatically derives multidimensional scoring rules aligned with skill execution from real behavioral trajectories, decoupling process quality from task outcomes across four dimensions: skill selection, adherence, composition, and reflection. This approach not only uncovers failure modes obscured by conventional accuracy metrics but also generates high-quality supervisory signals for process-level learning. Experimental results demonstrate that the evolved scoring criteria enable finer-grained evaluation and significantly outperform purely outcome-driven training methods in enhancing agents’ skill utilization capabilities.

This work addresses the unreliability of existing computer-use agent (CUA) trajectory validation methods, which distorts evaluation and training signals. To overcome this limitation, the authors propose CUAVerifier, a general-purpose validator grounded in four core principles: non-overlapping scoring criteria, separation of process- and outcome-based rewards, distinction between controllable and uncontrollable failures, and divide-and-conquer context management. Leveraging cascaded error-free failure analysis, a dual-signal mechanism, and a hybrid rule-model architecture, CUAVerifier enables fine-grained trajectory assessment on CUAVerifierBench—a newly constructed, multidimensional, human-annotated benchmark. Experimental results demonstrate that CUAVerifier achieves human-level agreement with expert judgments, exhibits near-zero false positive rates, and significantly outperforms baseline approaches such as WebVoyager and WebJudge.

Current human-AI collaborative agent systems treat skills as code that is blindly trusted without verification, necessitating human review for every irreversible action and thereby hindering scalability. This work proposes redefining skills explicitly as untrusted artifacts until verified, introducing a trust model, capability gating mechanisms, and a formally verifiable dual-condition correctness criterion that triggers human oversight only for unverified skills. We present a model- and framework-agnostic trust architecture, articulate a portable runtime specification grounded in ten design principles, and validate its efficacy through an open-source implementation. Our approach significantly enhances system scalability and sustainability without requiring model retraining or proprietary infrastructure.

This work addresses the growing challenge that while code-generating agents are becoming increasingly capable, their outputs are difficult to reliably verify against ambiguous human intent, and fixed reward signals are prone to exploitation. To tackle this, we propose a systematic evaluation framework for verification signals along three dimensions: scalability, faithfulness, and robustness—the first of its kind—and advocate for the co-evolution of verification and generation capabilities. We design four task-specific verifiers—test cases, scoring rules, user feedback, and automated agent-based validation—and conduct in-depth experiments to uncover both the efficacy and failure modes of reward signals. Our approach substantially mitigates reward hacking and consistently improves task completion quality across multiple internal and public benchmarks.

This study addresses the critical lack of systematic security auditing in the current ecosystem of AI agent skills, which harbors widespread yet underrecognized security risks. The authors present the first security taxonomy for agent skills grounded in real-world vulnerabilities and introduce SkillScan, a multi-stage detection framework that integrates static code analysis with large language model–based semantic classification. Empirical evaluation across 42,447 skills from two major marketplaces reveals that 26.1% contain vulnerabilities, with data leakage and privilege escalation being the most prevalent; executable-script skills exhibit significantly higher risk. The proposed method achieves 86.7% precision and 82.5% recall in vulnerability detection. The dataset and toolkit are publicly released to support further research.

This work investigates the phenomenon of “phantom guardrails,” wherein self-improving agents fabricate errors and apply ineffective safeguards in the absence of actual failures. To systematically examine this behavior, the authors construct a counterfactual hallucination laboratory—a deterministic, non-interventional environment—employing a large language model proposer, byte-precise oracle verification, deterministic micro-experimental setups, and controlled variable analysis. Their experiments reveal that when rule-like patterns, open-ended rule sets, and pre-specified failure instructions coexist, agents structurally generate spurious fixes in 15 out of 60 runs. This tendency persists across both single-proposal and iterative acceptance cycles. The study introduces the first reproducible evaluation framework for this issue, offering a novel dimension for assessing the reliability of self-improving systems.

Existing program verification approaches struggle to reliably identify skill deficiencies and extract actionable improvement signals from ambiguous feedback. This work proposes the first skill self-evolution framework tailored for program verification, which attributes verification failures to specific skill gaps, extracts diagnostic features as reusable experience, and selectively adopts only those skill revisions that enhance verification performance while preserving program semantics. Integrating large language model agents, verification trajectory analysis, diagnostic signature extraction, and an iterative optimization mechanism under semantic constraints, the framework consistently outperforms current baselines across diverse verification tools, agent architectures, and large model backends, thereby overcoming the limitations of traditional self-evolution methods in complex verification scenarios.

This work addresses the challenge faced by hardware LLM agents in accurately locating relevant RTL modules, testbenches, and dependencies within complex Verilog designs and recovering from sparse verification failures. To this end, it proposes a test-time scaling framework that requires no model fine-tuning, treating the agent’s natural language capabilities as evolvable strategies. Through an Oracle-Mutator-Selector loop integrated with execution trace mining and dense verification feedback, the framework dynamically generates task-specific skills to guide search, editing, and recovery processes. Notably, it introduces the first skill evolution mechanism driven by dense verification feedback, significantly improving the agent’s success rate on hard EDA tasks without updating model weights or relying on high-quality fine-tuning data, and successfully solves multiple previously unsolved complex Verilog design problems.

This work addresses the performance degradation often observed in self-improving agents due to a misalignment between self-generated validation signals and actual deployment performance. To mitigate this issue, the authors propose the Sealed External Audit Loop (SEAL) mechanism, which preserves the agent’s ability to generate its own test cases while incorporating an external audit signal that the agent cannot manipulate. SEAL enforces a fixed audit loop, compares candidate policies, conducts sealed evaluations, and maintains state consistency to effectively block harmful updates. Experimental results across six models and three random seeds demonstrate that SEAL significantly outperforms unprotected baselines, robustly alleviating self-validation failure—particularly in capability-stratified agents.

This work investigates the performance degradation of verifier-driven self-improving vision-language models in cross-task settings, where task-specific verifiers often hinder generalization. The study systematically demonstrates that even highly accurate verifiers can significantly impair student model performance—by 3.4 to 10.9 percentage points—when their task alignment is poor, and introduces a “directional mismatch” mechanism to explain this phenomenon. Building on frozen verifier scores, preference pair construction, and DPO training, the authors integrate variance analysis with a progress-gated replay strategy to derive practical guidelines for verifier selection at deployment time. Empirical validation on benchmarks such as MMMU confirms the findings, with consistent results reproduced across two Qwen-VL model variants.