automated diagnostic generation

Existing LLM-based automated program repair (APR) approaches predominantly rely on a single software artifact, neglecting systematic investigation and synergistic modeling of complementary multi-source artifacts—such as bug reports, stack traces, and debugging information. This work presents the first comprehensive analysis of the differential contributions of diverse software artifacts to fault localization and repair. We propose DEVLoRe, an end-to-end APR framework that integrates multi-source information via prompt engineering and context augmentation to jointly perform method-level and line-level fault localization and generate test-suite-passing patches. Evaluated on Defects4J v2.0, DEVLoRe achieves fault localization accuracy of 49.3% for single-fault bugs and 47.6% for multi-fault bugs. It generates plausible, test-suite-passing patches at rates of 56.0% and 14.5%, respectively—marking substantial improvements over state-of-the-art methods.

Bug fixing is a complex and time-consuming task in software development. Bug localization research tends to focus on the accuracy of automated tools that suggest source code files for developers to look at. However, little is known about how developers use these tools in practice. This paper reports on an ongoing qualitative user study. Eleven participants worked through four realistic bug localization tasks in a controlled environment and were given varying levels of support information offered by a specialized tool. Participants were asked to think aloud in a semi-structured interview session. The preliminary findings provide insight into three aspects of practice: how developers interact with tools, the role social and contextual information plays, and problem solving. The study demonstrates that bug localization is complex and suggests that the adoption of effective tools depends on more than their accuracy.

Software evolution frequently causes test failures, leading to high maintenance costs and low efficiency. This paper proposes TaRGet, the first framework to formalize test repair as a context-aware code translation task. Leveraging pre-trained models such as CodeT5 and CodeLlama, TaRGet automates repair via failure-context extraction, error-pattern-aware input construction, and two-stage fine-tuning. Its key contributions are: (1) a novel formalization of test repair as translation; (2) TaRBench—the first large-scale, empirically grounded benchmark comprising over 45K real-world test repairs; and (3) a reliability-prediction guidance mechanism, empirically validated for cross-project cold-start generalization. On TaRBench, TaRGet achieves a 66.1% exact-match repair rate—significantly outperforming state-of-the-art baselines—without requiring any project-specific training data.

This work addresses the robustness evaluation of learning-based defect prediction and program repair models. We propose a novel method for synthesizing highly stealthy (hard-to-detect) and highly complex (hard-to-fix) defects. Our approach innovatively leverages large language models’ (LLMs) attention mechanisms to guide multi-point code mutation, enabling precise semantic-preserving perturbations while maintaining both functional and representational similarity. Integrating program synthesis with large-scale mutant generation, we construct a high-quality dataset comprising over 435,000 synthetic defects. Experimental results demonstrate that our defects increase the false-negative rate of mainstream defect predictors by 37% and raise the repair failure rate of state-of-the-art program repair models by 52%, significantly outperforming existing baselines. To our knowledge, this is the first work to deeply integrate LLM attention analysis into defect synthesis, establishing a scalable and interpretable paradigm for model robustness assessment.

This work addresses the challenge that code generated by large language models often contains defects, yet existing feedback signals are too coarse-grained to precisely localize errors. To this end, the authors propose FLARE, a novel framework that introduces lightweight line-level suspiciousness prediction and a multi-candidate region search mechanism for the first time. By leveraging execution outcomes to rank and select among repair candidates, FLARE enables fine-grained, iterative code refinement. Experimental results on LiveCodeBench and BigCodeBench demonstrate that FLARE outperforms the strongest baseline by 1.72%–7.42% on average. Moreover, employing a 10-candidate search strategy yields an additional 8.50% improvement over single-candidate approaches, substantially enhancing the accuracy and effectiveness of code repair.

This work addresses the limitation of existing large language model–based automated program repair approaches, which rely on end-to-end test feedback and struggle to precisely identify internal logical deviations. To overcome this, the authors propose SpecTune, a framework that inserts checkpoints along execution paths to generate localized postconditions and evaluates intermediate program behaviors against dynamic execution results, thereby providing fine-grained debugging signals. SpecTune introduces an intermediate behavior reasoning mechanism and designs two key signals—a specification validation signal (α) and a discriminative signal (β)—to substantially enhance the reliability of automatically generated specifications and the precision of repairs. Experimental results demonstrate that SpecTune significantly outperforms current baseline methods in both fault localization accuracy and repair success rate.

This study presents the first systematic investigation into how code overviews generated by open-source large language models (LLMs) influence developers’ experience, trust calibration, and efficiency when debugging unfamiliar codebases. Leveraging 26 real-world Java vulnerabilities, the authors construct an automated pipeline for generating and evaluating LLM-produced overviews. Through think-aloud experiments with 26 developers and multi-author qualitative coding, they analyze developer preferences and behaviors. The findings reveal that key overview attributes—such as granularity of detail, scannability, and guiding tone—significantly impact usability. Moreover, developers tend to overestimate the reliability of AI-generated content, while automated LLM-based evaluations frequently exhibit flattery, hallucination, and inconsistency, underscoring the critical need for human-in-the-loop assessment.

This study addresses a critical yet previously underexplored issue in large language model (LLM)-driven software development: the contamination of automatically generated tests by erroneous code. The authors systematically uncover and empirically validate this error propagation phenomenon, demonstrating that when tests are generated based on incorrect code within multi-step agent workflows—across diverse programming tasks and various prompting strategies, including chain-of-thought—the resulting tests exhibit significantly lower defect detection rates (14%) compared to independently generated tests (25%). These findings challenge the prevailing assumption that LLM-generated tests can serve as reliable, independent oracles, thereby highlighting the substantial risk of test bias in LLM-augmented development pipelines.

This work addresses a critical yet overlooked issue in large language model (LLM)-based unit test generation: the tendency to produce “misleading tests” that validate incorrect behavior rather than expose bugs in defective code. The study presents the first quantitative characterization of this “misleading effect,” revealing the dual negative impact of code defects on test generation quality. To mitigate this, the authors propose a novel specification-driven paradigm that replaces the original (potentially buggy) code with model-generated docstrings as prompts, thereby circumventing defect-induced biases. Integrating prompt engineering, automated specification synthesis, and a multi-round feedback mechanism, the approach significantly reduces misleading tests and substantially improves effective bug detection rates—demonstrating consistent gains in both defective and non-defective code scenarios.