Problem
Research questions and friction points this paper is trying to address.
coding agents
human-AI interaction
software engineering
prompt analysis
developer understanding
Beyond the Prompt: Linking What Developers Ask, Do, and Understand with Coding Agents
🤖 AI Summary
This study addresses the limitation that analyzing prompts alone is insufficient for comprehensively evaluating developer interactions with AI programming agents. To overcome this, we propose a novel multidimensional interaction analysis framework termed "Say-Do-Understand," which integrates prompt data, screen activity, and comprehension metrics through a systematic five-stage end-to-end workflow. Employing an observational methodology, the analysis utilizes a prompt codebook, a screen activity coding scheme, and dual scoring rubrics. An empirical study involving ten experienced developers validates the proposed approach. Furthermore, four developer personas synthesizing task performance and comprehension levels are introduced to elucidate behavioral variations. Notably, the findings reveal that excessive reliance on agent self-checking significantly reduces developers' autonomous testing time.
📝 Abstract
Coding agents can now change code for developers, who describe goals, supply context, and respond to the agent's work. Yet prompts, screen activity, and task success each tell only part of this story. We present Say, Do, Understand, an end-to-end workflow for analyzing what developers write to an agent, what they do while it works, and what they can explain afterwards. The workflow has five stages (Capture, Prepare, Analyze, Integrate, and Interpret) and three instruments: a prompt codebook, a scheme for coding screen-recorded activities and events, and separate rubrics for explaining the process and the solution. We applied it in an observational study of ten experienced developers who used GitHub Copilot on an unfamiliar codebase. Crossing task performance with understanding produced four personas. The two measures agreed for eight developers but split for two: one passed most tests but could not explain the solution, and another passed few tests but explained it well. In this sample, the personas that most often asked the agent to check its work spent the least time testing on their own. These patterns are descriptive and do not generalize beyond the sample. We recommend the workflow to computing educators, industry practitioners, and researchers to adapt and evaluate human--AI communication in software engineering.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Coding Agents
End-to-End Workflow
Human-AI Interaction
Developer Personas
Software Engineering
🔎 Similar Papers
No similar papers found.
