Enhancing Software Development with Context-Aware Conversational Agents: A User Study on Developer Interactions with Chatbots

2025-05-13

AI Summary

This study investigates software developers’ functional requirements for large language model (LLM)-driven conversational agents to enhance development efficiency and user experience while supporting differentiated interactions for novices and experts. Through user behavior analysis, semi-structured interviews, and contextualized task evaluations, we systematically identify three core requirements: multi-level experience adaptation, history-aware interaction, and deep code-context understanding—the first empirical characterization of such needs. Based on these findings, we propose design principles for context-aware (CA) conversational agents tailored to software engineering and implement a prototype to validate feasibility. We prioritize three high-impact capabilities: task automation, Git version-control integration, and experience-level–adaptive response generation. Our work delivers the first empirically grounded design guidelines for intelligent programming assistants, addressing critical gaps in human-AI collaborative programming—specifically, requirement modeling and context-adaptive interaction mechanisms.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: User Experience and Usability

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Software development is a cognitively intensive process requiring multitasking, adherence to evolving workflows, and continuous learning. With the rise of large language model (LLM)-based tools, such as conversational agents (CAs), there is growing interest in supporting developers through natural language interaction. However, little is known about the specific features developers seek in these systems. We conducted a user study with 29 developers using a prototype text-based chatbot to investigate preferred functionalities. Our findings reveal strong interest in task automation, version control support, and contextual adaptability, especially the need to tailor assistance for both novice and experienced users. We highlight the importance of deep contextual understanding, historical interaction awareness, and personalized support in CA design. This study contributes to the development of context-aware chatbots that enhance productivity and satisfaction, and it outlines opportunities for future research on human-AI collaboration in software engineering.

Problem

• Identifying developer-preferred features in LLM-based conversational agents
• Exploring task automation and version control support needs in chatbots
• Investigating contextual adaptability for novice and experienced developers

Innovation

• Context-aware chatbots for developer support
• Task automation and version control integration
• Personalized assistance for all skill levels