Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks

2025-10-13

AI Summary

Little is known about how developers practically integrate AI-powered code generation tools—such as Amazon CodeWhisperer—into real-world programming workflows. Method: We conducted two mixed-methods user studies, combining controlled experiments, a custom telemetry plugin for fine-grained interaction logging, and qualitative thematic analysis. Contribution/Results: We identify four novel behavioral patterns: (1) incremental code refinement, (2) explicit natural-language prompting via comments, (3) structured baseline construction grounded in model suggestions, and (4) synergistic tool usage integrating external resources (e.g., documentation, Stack Overflow). This work provides the first empirical characterization of multi-layered human–AI collaboration in AI-assisted programming. It extends theoretical frameworks of developer–AI interaction and yields actionable, evidence-based design implications—particularly for prompt guidance, suggestion integration mechanisms, and context-awareness—in next-generation AI coding assistants.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

The use of AI code-generation tools is becoming increasingly common, making it important to understand how software developers are adopting these tools. In this study, we investigate how developers engage with Amazon's CodeWhisperer, an LLM-based code-generation tool. We conducted two user studies with two groups of 10 participants each, interacting with CodeWhisperer - the first to understand which interactions were critical to capture and the second to collect low-level interaction data using a custom telemetry plugin. Our mixed-methods analysis identified four behavioral patterns: 1) incremental code refinement, 2) explicit instruction using natural language comments, 3) baseline structuring with model suggestions, and 4) integrative use with external sources. We provide a comprehensive analysis of these patterns .

Problem

• Analyzing developers' interaction patterns with AI code-generation tools
• Investigating behavioral patterns during programming tasks using CodeWhisperer
• Understanding how software developers adopt AI programming assistants

Innovation

• Conducted user studies with CodeWhisperer telemetry plugin
• Identified four behavioral patterns in developer interactions
• Analyzed incremental refinement and natural language instructions