automated program repair

Large language models can generate runnable software artifacts, but their security remains difficult to evaluate end to end. This study examines that problem through a Detect--Repair--Verify (DRV) workflow, in which vulnerabilities are detected, repaired, and then rechecked with security and functional tests. It addresses four gaps in current evidence: the lack of test-grounded benchmarks for LLM-generated artifacts, limited evidence on pipeline-level effectiveness, unclear reliability of detection reports as repair guidance, and uncertain repair trustworthiness under verification. To support this study, EduCollab is constructed as a multi-language, multi-granularity benchmark of runnable LLM-generated web applications in PHP, JavaScript, and Python. Each artifact is paired with executable functional and exploit test suites, and the benchmark spans project-, requirement-, and file-level settings. On this benchmark, the study compares unrepaired baselines, single-pass detect--repair, and bounded iterative DRV under comparable budget constraints. Outcomes are measured by secure-and-correct yield, and intermediate artifacts and iteration traces are analyzed to assess report actionability and repair failure modes. The results show that bounded iterative DRV can improve secure-and-correct yield over single-pass repair, but the gains are uneven at the project level and become clearer at narrower repair scopes. Detection reports are often useful for downstream repair, but their reliability is inconsistent. Repair trustworthiness also depends strongly on repair scope. These findings highlight the need for test-grounded, end-to-end evaluation of LLM-based vulnerability management workflows.

This work addresses the industrial ineffectiveness of automated program repair (APR) and the misalignment between academic benchmarks and real-world defects. We propose a novel approach that leverages formal specifications—expressed in the Java Modeling Language (JML)—to guide large language models (LLMs) in generating high-coverage unit tests. By embedding specifications into the test-generation pipeline and integrating contract automata with human feedback, our method significantly improves detection of logic errors, string manipulations, and exception-handling edge cases. Evaluation shows that our technique outperforms state-of-the-art APR methods on standard benchmarks; JML specifications prove particularly valuable for exposing complex semantic errors; yet passing generated tests does not guarantee repair correctness. The study reveals a fundamental gap between academic evaluation criteria and industrial requirements, offering both a scalable technical pathway and empirical evidence toward practical, specification-guided program repair.

Existing LLM-based automated program repair (APR) approaches predominantly rely on a single software artifact, neglecting systematic investigation and synergistic modeling of complementary multi-source artifacts—such as bug reports, stack traces, and debugging information. This work presents the first comprehensive analysis of the differential contributions of diverse software artifacts to fault localization and repair. We propose DEVLoRe, an end-to-end APR framework that integrates multi-source information via prompt engineering and context augmentation to jointly perform method-level and line-level fault localization and generate test-suite-passing patches. Evaluated on Defects4J v2.0, DEVLoRe achieves fault localization accuracy of 49.3% for single-fault bugs and 47.6% for multi-fault bugs. It generates plausible, test-suite-passing patches at rates of 56.0% and 14.5%, respectively—marking substantial improvements over state-of-the-art methods.

This work addresses the challenge of automated repair for declarative formal specifications—exemplified by Alloy—where existing automated program repair (APR) techniques predominantly target imperative languages and lack effective support for declarative specifications. We propose the first dual-agent LLM framework tailored for declarative specification repair: a Repair Agent that generates fixes and a Prompt Agent that dynamically refines prompts and integrates feedback from formal verification. Leveraging models such as GPT-4, combined with advanced prompt engineering and collaborative reasoning, our approach achieves substantial performance gains on standard Alloy benchmarks—significantly outperforming state-of-the-art specialized APR tools in repair success rate. Empirical evaluation demonstrates that large language models exhibit strong generalization capability and practical utility in formal specification repair, providing the first systematic validation of LLMs’ effectiveness and superiority in this domain.

Existing LLM-based program repair methods suffer from misalignment between pretraining objectives and the repair task, and rely on manual or tool-assisted error localization—leading to suboptimal knowledge utilization and constrained repair search spaces. This paper proposes *direct whole-program debugging*, a paradigm that abandons the conventional two-stage “localize-then-repair” pipeline and introduces *objective alignment*: it uniformly models repair as an autoregressive code continuation task. We design D4C, a lightweight prompting framework for decoder-only LLMs (e.g., GPT-4), which integrates test-case context to enable end-to-end function-level generation and sampling-based optimization. Evaluated on Defects4J, D4C repairs 180 bugs with only 10 samples per attempt—outperforming state-of-the-art methods under perfect localization by 10% and improving sampling efficiency by 90%.

Existing large code models struggle to generate executable intermediate formal specifications, limiting precise verification and repair of program behavioral errors. This work proposes SpecCoder, a novel framework that focuses on generating executable inline assertions at critical program locations, thereby transforming static annotations into verifiable evidence. SpecCoder employs verification-guided training, fine-tuning the Qwen2.5-Coder series models using correct programs, behavioral mutants, and multi-round specification refinement trajectories. Evaluated on the HumanExec benchmark, SpecCoder substantially improves the correctness (+55.8%), completeness (+358.1%), and assertion validity (+26.6%) of inline specifications, significantly enhancing program verification and repair capabilities.

This work addresses the limitation of existing large language model–based automated program repair approaches, which rely on end-to-end test feedback and struggle to precisely identify internal logical deviations. To overcome this, the authors propose SpecTune, a framework that inserts checkpoints along execution paths to generate localized postconditions and evaluates intermediate program behaviors against dynamic execution results, thereby providing fine-grained debugging signals. SpecTune introduces an intermediate behavior reasoning mechanism and designs two key signals—a specification validation signal (α) and a discriminative signal (β)—to substantially enhance the reliability of automatically generated specifications and the precision of repairs. Experimental results demonstrate that SpecTune significantly outperforms current baseline methods in both fault localization accuracy and repair success rate.

This work addresses a critical yet overlooked reliability issue in code generated by large language models (LLMs): despite passing compilation and unit tests, such code often fails in deployment due to structural inconsistencies—such as missing configurations, invalid imports, or omitted security controls—that evade detection by conventional CI/SAST tools. The paper introduces the “patchwork problem” to characterize these cross-module global defects, proposes an eight-category taxonomy specific to LLM-generated code, and formalizes structural consistency via invariants derived from a multidimensional code graph encompassing imports, calls, dependencies, configurations, and routing. Building on this foundation, the authors design a hybrid verification framework that integrates traditional static analysis with custom graph-based invariant checkers to precisely identify structural flaws invisible to existing tools. Empirical evaluation reveals that such defects are pervasive across major LLMs under diverse prompting strategies and exhibit distinct model-specific patterns.

This work addresses the challenge of poor generalizability in dynamically inferred specifications, which often stems from insufficient test coverage and necessitates extensive manual filtering. To mitigate this issue, the study introduces a novel integration of large language model (LLM)-generated counterexample tests into the dynamic inference pipeline. By leveraging tools such as SpecFuzzer to automatically validate the inferred assertions, the approach significantly improves precision without compromising recall. Experimental results demonstrate that the method effectively eliminates up to 11.68% of invalid assertions, achieving a maximum precision gain of 7% in specification inference. This advancement enhances both the accuracy and automation level of dynamic specification inference, reducing reliance on human intervention while maintaining robustness in inferred program specifications.

This work addresses the limited diversity in repair strategies generated by current large language models for automated program repair, which often stems from redundant execution traces and repetitive sampling. To overcome this, the authors propose CT-Repair, a novel framework that integrates static and dynamic evidence by combining Code Property Graphs (CPGs) with Temporal Execution Graphs (TEGs). CT-Repair introduces a finite state machine–guided multi-perspective agent collaboration mechanism, enabling independent generation and optimization of diverse repair strategies. Coupled with a three-stage filtering pipeline and validation feedback, the approach substantially enhances both repair diversity and accuracy. Evaluated on 854 Java bugs from Defects4J v3.0, CT-Repair successfully repairs 489, outperforming ReinFix and RepairAgent; the joint use of three perspectives yields 99 more fixes than the strongest single perspective, while execution-based filtering reduces the search space by an average of 94.85%.