What Makes a GitHub Issue Ready for Copilot?

2025-12-24

AI Summary

GitHub issue quality significantly influences the mergeability of code generated by AI coding agents (e.g., GitHub Copilot), yet current best practices remain vague and lack empirical validation. Method: We systematically construct 32 fine-grained, interpretable issue quality guidelines and propose the first explainable, AI-aware issue mergeability prediction model. Leveraging text-based feature engineering and machine learning, our model achieves a median AUC of 72%. Contribution/Results: We identify key determinants of mergeability: issues that are concise, narrowly scoped, include implementation guidance, and specify code locations exhibit higher merge rates; conversely, issues involving external dependencies or configuration yield significantly lower merge rates. Our work provides both theoretical foundations and practical tools—namely, actionable quality criteria and an interpretable predictive model—to enhance human–AI collaborative software development efficacy.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

AI-agents help developers in different coding tasks, such as developing new features, fixing bugs, and reviewing code. Developers can write a Github issue and assign it to an AI-agent like Copilot for implementation. Based on the issue and its related discussion, the AI-agent performs a plan for the implementation, and executes it. However, the performance of AI-agents and LLMs heavily depends on the input they receive. For instance, a GitHub issue that is unclear or not well scoped might not lead to a successful implementation that will eventually be merged. GitHub Copilot provides a set of best practice recommendations that are limited and high-level. In this paper, we build a set of 32 detailed criteria that we leverage to measure the quality of GitHub issues to make them suitable for AI-agents. We compare the GitHub issues that lead to a merged pull request versus closed pull request. Then, we build an interpretable machine learning model to predict the likelihood of a GitHub issue resulting in a merged pull request. We observe that pull requests that end up being merged are those originating from issues that are shorter, well scoped, with clear guidance and hints about the relevant artifacts for an issue, and with guidance on how to perform the implementation. Issues with external references including configuration, context setup, dependencies or external APIs are associated with lower merge rates. We built an interpretable machine learning model to help users identify how to improve a GitHub issue to increase the chances of the issue resulting in a merged pull request by Copilot. Our model has a median AUC of 72%. Our results shed light on quality metrics relevant for writing GitHub issues and motivate future studies further investigate the writing of GitHub issues as a first-class software engineering activity in the era of AI-teammates.

Problem

• Identifies criteria for GitHub issue quality for AI agents
• Predicts merge likelihood of pull requests from issues
• Provides guidance to improve issues for successful AI implementation

Innovation

• Develops 32 detailed criteria for GitHub issue quality
• Builds interpretable ML model to predict merge likelihood
• Identifies key issue attributes affecting AI-agent success