Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

2026-08-05

AI Summary

Existing software engineering benchmarks rely on high-quality issue reports, making it difficult to evaluate the ability of coding agents to proactively identify and fix bugs in real-world scenarios. This work proposes the first benchmark that operates without issue reports, enabling evaluation of proactive bug localization and repair across multiple defects and eight programming languages (1,663 tasks). The benchmark introduces a novel difficulty-aware task construction pipeline and a dual-track evaluation framework, shifting the assessment paradigm from passive repair to active discovery for the first time. Experimental results demonstrate that current state-of-the-art coding agents exhibit limited performance under this setting, underscoring the benchmark’s challenge and necessity.

Abstract

Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase. However, existing SWE benchmarks typically assume that high-quality issue reports with detailed information are always available, which is easily violated in practice due to the complexity of report acquisition and curation. To address this, we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. Beyond shifting the focus from existing reactive bug fixing to proactive bug fixing, Active-SWE enables a more in-depth evaluation by expanding the scope from fixing a specific recorded bug to multiple-bug fixing and potential bug discovery scenarios. To construct Active-SWE, we propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating comprehensive evaluation of proactive bug-fixing capability. Extensive experiments reveal that most state-of-the-art coding agents struggle with proactive bug-fixing tasks, demonstrating limited performance in locating and resolving recorded bugs, handling multiple bug fixing scenarios, and discovering valid potential bugs.

Problem

• proactive bug fixing
• coding agents
• software engineering
• issue reports
• bug discovery

Innovation

• proactive bug fixing
• coding agents
• LLM-based software engineering
• benchmarking
• multi-bug discovery