Do Comments and Expertise Still Matter? An Experiment on Programmers' Adoption of AI-Generated JavaScript Code

2025-03-14

AI Summary

This study investigates how AI-generated annotations and developer experience influence programmers’ adoption of AI-recommended JavaScript code. We conducted an online experiment (N=173) using LeetCode programming tasks and an AST-based, objective code similarity metric to quantify adoption behavior—replacing subjective self-reports with a reproducible, behavioral measure for the first time. Results show that AI-generated annotations significantly increase code adoption rates (p<0.001), and this effect is robust across junior to senior developers, with no moderation by experience level. Our contributions are threefold: (1) establishing the first objective adoption evaluation paradigm grounded in structural code similarity; (2) empirically demonstrating annotations as a critical trust mediator between AI recommendations and human adoption; and (3) providing evidence-based design principles for AI programming assistants—particularly underscoring the value of explanatory annotations in fostering reliable human-AI collaboration.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: User Experience and Usability
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

This paper investigates the factors influencing programmers' adoption of AI-generated JavaScript code recommendations. It extends prior research by (1) utilizing objective (as opposed to the typically self-reported) measurements for programmers' adoption of AI-generated code and (2) examining whether AI-generated comments added to code recommendations and development expertise drive AI-generated code adoption. We tested these potential drivers in an online experiment with 173 programmers. Participants were asked to answer some questions to demonstrate their level of development expertise. Then, they were asked to solve a LeetCode problem without AI support. After attempting to solve the problem on their own, they received an AI-generated solution to assist them in refining their solutions. The solutions provided were manipulated to include or exclude AI-generated comments (a between-subjects factor). Programmers' adoption of AI-generated code was gauged by code similarity between AI-generated solutions and participants' submitted solutions, providing a more reliable and objective measurement of code adoption behaviors. Our findings revealed that the presence of comments significantly influences programmers' adoption of AI-generated code regardless of the participants' development expertise.

Problem

• Factors influencing AI-generated JavaScript code adoption
• Impact of AI-generated comments on code adoption
• Role of development expertise in AI code acceptance

Innovation

• Objective measurement of AI-generated code adoption
• AI-generated comments impact code adoption
• Online experiment with 173 programmers