The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot

2024-10-02

Social Science Research Network

AI Summary

This study investigates the causal mechanisms through which generative AI tools (e.g., GitHub Copilot) influence open-source collaborative development. Leveraging large-scale behavioral logs from GitHub repositories, we apply generalized synthetic control methods and rigorous causal inference models to estimate treatment effects. Our analysis yields three key contributions: First, AI-augmented pair programming significantly enhances project-level productivity by 6.5%, while increasing individual developer productivity and participation rates by 5.5% and 5.4%, respectively. Second, code quality—measured via static analysis, test coverage, and post-merge defect rates—shows no statistically significant decline, indicating efficiency gains do not compromise quality. Third, we identify a novel trade-off: pull request integration time increases by 41.6%, revealing heightened coordination costs in AI-mediated collaboration. To our knowledge, this is the first study to provide causal evidence and a theoretical framework characterizing the dual effects—both productive and coordinative—of generative AI on open-source software development.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Reasoning under Uncertainty: Causality

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Generative artificial intelligence (AI) has opened the possibility of automated content production, including coding in software development, which can significantly influence the participation and performance of software developers. To explore this impact, we investigate the role of GitHub Copilot, a generative AI pair programmer, on software development in open-source community, where multiple developers voluntarily collaborate on software projects. Using GitHub's dataset for open-source repositories and a generalized synthetic control method, we find that Copilot significantly enhances project-level productivity by 6.5%. Delving deeper, we dissect the key mechanisms driving this improvement. Our findings reveal a 5.5% increase in individual productivity and a 5.4% increase in participation. However, this is accompanied with a 41.6% increase in integration time, potentially due to higher coordination costs. Interestingly, we also observe the differential effects among developers. We discover that core developers achieve greater project-level productivity gains from using Copilot, benefiting more in terms of individual productivity and participation compared to peripheral developers, plausibly due to their deeper familiarity with software projects. We also find that the increase in project-level productivity is accompanied with no change in code quality. We conclude that AI pair programmers bring benefits to developers to automate and augment their code, but human developers' knowledge of software projects can enhance the benefits. In summary, our research underscores the role of AI pair programmers in impacting project-level productivity within the open-source community and suggests potential implications for the structure of open-source software projects.

Problem

• Impact of GitHub Copilot on open-source collaboration
• Tradeoff between increased code contributions and coordination time
• Differential effects on core vs peripheral developers

Innovation

• GitHub Copilot boosts code contributions by 5.9%
• AI pair programmers increase coordination time by 8%
• Core and peripheral developers show differential impacts