Prompting in Practice: Investigating Software Developers' Use of Generative AI Tools

2025-10-07

AI Summary

Despite growing adoption of generative AI in software development, its integration into professional practice—particularly across diverse engineering tasks—remains poorly understood. Method: We employed a mixed-methods approach, combining a large-scale survey of 91 professional software engineers with in-depth qualitative analysis, grounded in a software engineering task taxonomy to systematically examine prompting strategies, multi-turn interaction patterns, and reliability assessment behaviors. Contribution/Results: We find that while code generation is widely adopted, proficiency differentiation emerges more clearly in advanced tasks such as debugging and code review; documentation generation achieves the highest reliability, whereas complex logic implementation remains challenging. We propose a novel, empirically grounded “progressive workflow integration” paradigm—from isolated code generation toward deep, context-aware toolchain integration—and provide the first evidence-based characterization of developers’ iterative, multi-turn interaction preferences with generative AI in real-world settings, offering concrete empirical benchmarks and actionable design guidelines for AI-assisted development tools.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageHumans and AI: Game Design — Procedural Content Generation & StorytellingSearch and Optimization: Sampling/Simulation-based Search

Application Category

Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAISocial Networks and Social Media: Generative AI / large language models and their impact on social systemsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The integration of generative artificial intelligence (GenAI) tools has fundamentally transformed software development. Although prompt engineering has emerged as a critical skill, existing research focuses primarily on individual techniques rather than software developers' broader workflows. This study presents a systematic investigation of how software engineers integrate GenAI tools into their professional practice through a large-scale survey examining prompting strategies, conversation patterns, and reliability assessments across various software engineering tasks. We surveyed 91 software engineers, including 72 active GenAI users, to understand AI usage patterns throughout the development process. Our 14 key findings show that while code generation is nearly universal, proficiency strongly correlates with using AI for more nuanced tasks such as debugging and code review, and that developers prefer iterative multi-turn conversations to single-shot prompting. Documentation tasks are perceived as most reliable, while complex code generation and debugging present sizable challenges. Our insights provide an empirical baseline of current developer practices, from simple code generation to deeper workflow integration, with actionable insights for future improvements.

Problem

• Investigating how developers integrate generative AI tools into software workflows
• Analyzing prompting strategies and conversation patterns in engineering tasks
• Assessing reliability challenges in code generation and debugging processes

Innovation

• Surveyed software engineers' GenAI integration practices
• Analyzed multi-turn conversations over single-shot prompting
• Empirically assessed reliability across development task types