Technology Category
Humans and AI: Human-Aware Planning and Behavior PredictionNatural Language Processing: Code Generation / Program Synthesis from Natural LanguageCognitive Modeling & Cognitive Systems: Simulating Human Behavior
The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
🤖 AI Summary
This study investigates the impact of GitHub Copilot on undergraduate computer science students performing brownfield programming—extending unfamiliar legacy codebases with new functionality. Employing a controlled experiment, the research integrates performance logging, behavioral trace data, and structured exit interviews. Results demonstrate that Copilot significantly improves programming efficiency: task completion time decreases by 35%, and solution progress increases by 50%; manual coding time declines by 11% and web searching by 12%. Crucially, the study identifies a paradigm shift in programming behavior and reveals students’ widespread lack of understanding of how AI-generated suggestions are produced—highlighting “explanability” as a core pedagogical challenge in human-AI collaboration. This work provides the first empirically grounded efficacy evaluation framework for AI programming assistants in education, along with actionable insights for curriculum design and instructional practice.
Technology Category
Application Category
📝 Abstract
When graduates of computing degree programs enter the software industry, they will most likely join teams working on legacy code bases developed by people other than themselves. In these so-called brownfield software development settings, generative artificial intelligence (GenAI) coding assistants like GitHub Copilot are rapidly transforming software development practices, yet the impact of GenAI on student programmers performing brownfield development tasks remains underexplored. This paper investigates how GitHub Copilot influences undergraduate students' programming performance, behaviors, and understanding when completing brownfield programming tasks in which they add new code to an unfamiliar code base. We conducted a controlled experiment in which 10 undergraduate computer science students completed highly similar brownfield development tasks with and without Copilot in a legacy web application. Using a mixed-methods approach combining performance analysis, behavioral analysis, and exit interviews, we found that students completed tasks 35% faster (p<0.05) and made 50% more solution progress p (<0.05) when using Copilot. Moreover, our analysis revealed that, when using Copilot, students spent 11% less time manually writing code (p<0.05), and 12% less time conducting web searches (p<0.05), providing evidence of a fundamental shift in how they engaged in programming. In exit interviews, students reported concerns about not understanding how or why Copilot suggestions work. This research suggests the need for computing educators to develop new pedagogical approaches that leverage GenAI assistants' benefits while fostering reflection on how and why GenAI suggestions address brownfield programming tasks. Complete study results and analysis are presented at https://ghcopilot-icer.github.io/.
Problem
Research questions and friction points this paper is trying to address.
Impact of GitHub Copilot on student programming in legacy code
Effect of Copilot on coding speed and solution progress
Student concerns about understanding Copilot's suggestions in brownfield tasks
Innovation
Methods, ideas, or system contributions that make the work stand out.
GitHub Copilot boosts programming speed significantly
Reduces manual coding and web search time
Requires new teaching methods for GenAI
🔎 Similar Papers
No similar papers found.
