From Issues to Insights: RAG-based Explanation Generation from Software Engineering Artifacts

2026-01-09

arXiv.org

AI Summary

This work proposes the first application of Retrieval-Augmented Generation (RAG) to software engineering by leveraging problem-tracking data—such as GitHub Issues—to automatically generate context-aware explanations for software artifacts. Addressing the challenge that modern software systems are often poorly documented, with traditional documentation frequently outdated or incomplete, the approach enhances system transparency and user trust by delivering accurate, contextually relevant insights derived from structured development knowledge. The method integrates open-source tools with large language models to construct an end-to-end RAG pipeline, which, in empirical evaluation on representative projects, produces explanations aligning with human-written ones at a 90% fidelity rate. The results demonstrate high faithfulness, strong adherence to instructions, and reliable grounding in source evidence, thereby extending the frontier of explainable AI into the domain of software engineering.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The increasing complexity of modern software systems has made understanding their behavior increasingly challenging, driving the need for explainability to improve transparency and user trust. Traditional documentation is often outdated or incomplete, making it difficult to derive accurate, context-specific explanations. Meanwhile, issue-tracking systems capture rich and continuously updated development knowledge, but their potential for explainability remains untapped. With this work, we are the first to apply a Retrieval-Augmented Generation (RAG) approach for generating explanations from issue-tracking data. Our proof-of-concept system is implemented using open-source tools and language models, demonstrating the feasibility of leveraging structured issue data for explanation generation. Evaluating our approach on an exemplary project's set of GitHub issues, we achieve 90% alignment with human-written explanations. Additionally, our system exhibits strong faithfulness and instruction adherence, ensuring reliable and grounded explanations. These findings suggest that RAG-based methods can extend explainability beyond black-box ML models to a broader range of software systems, provided that issue-tracking data is available - making system behavior more accessible and interpretable.

Problem

• software explainability
• issue-tracking systems
• context-specific explanations
• software transparency
• development knowledge

Innovation

• Retrieval-Augmented Generation
• Explainability
• Issue-Tracking Systems
• Software Engineering Artifacts
• Explanation Generation