Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair

2025-10-03

AI Summary

Industrial-scale automated program repair (APR) systems suffer from high patch noise and impose substantial review overhead on developers. Method: This paper proposes a dual-LLM collaboration strategy: (1) a defect abstention mechanism that dynamically identifies and refrains from repairing intractable bugs, and (2) a patch semantic verification model that evaluates patch quality and filters out low-confidence repairs. Contribution/Results: Our approach is the first to integrate classification-based defect filtering with reasoning-driven patch validation, synergistically combining defect feature analysis and deep semantic understanding of large language models. Evaluated on Google’s real-world bug dataset, it achieves up to a 39-percentage-point improvement in single-attempt repair success rate. It significantly reduces the number of spurious patches presented to developers, shortens manual review time, and enhances developer trust in automated repair outputs.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry, but ultimately agent-generated patches still need to be reviewed by a human before committing them to ensure they address the bug. Showing unlikely patches to developers can lead to substantial noise, wasting valuable developer time and eroding trust in automated code changes. We introduce two complementary LLM-based policies to reduce such noise: bug abstention and patch validation policies. Bug abstention excludes bugs that the agentic APR system is unlikely to fix. Patch validation rejects patches that are unlikely to be a good fix for the given bug. We evaluate both policies on three sets of bugs from Google's codebase, and their candidate patches generated by an internal agentic APR system. On a set of 174 human-reported bugs, removing bugs and patch trajectories rejected by our policies can raise success rates by up to 13 percentage points and 15 percentage points, respectively, and by up to 39 percentage points in combination. On null pointer exceptions and sanitizer-reported bugs with machine-generated bug reports, patch validation also improves average single-sample success rates. This two-policy approach provides a practical path to the reliable, industrial-scale deployment of agentic APR systems.

Problem

• Reducing noise from unlikely patches in automated program repair systems
• Filtering bugs that agentic APR systems are unlikely to fix
• Validating patches to ensure they properly address reported bugs

Innovation

• Bug abstention excludes unlikely fixable bugs
• Patch validation rejects unlikely correct patches
• Dual-LLM policies combine to reduce developer noise