Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review

2024-09-18

ACM Computing Surveys

AI Summary

This paper addresses the practical challenges hindering the adoption of generative-AI-driven conversational bots and intelligent agents in software engineering (SE), where real-world deployment often yields suboptimal outcomes or introduces new risks. To systematically investigate this research-practice gap, the authors conduct a multi-source literature review—integrating peer-reviewed academic papers and industrial reports—employing thematic modeling and cross-source comparative analysis. They propose the first SE-specific taxonomy for bot applications and introduce a novel “Motivation–Challenge–Mitigation” tri-dimensional framework. This framework identifies seven core technical and organizational challenges, synthesizes twelve empirically grounded best practices, and prioritizes five high-impact research directions. The key innovation lies in bridging the theory-practice divide: by articulating actionable mitigation strategies and concrete translation pathways, the work provides both theoretical grounding and pragmatic guidance for effective bot deployment in SE contexts.

Technology Category

• Intelligent Robots: Other Foundations and Applications
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Bots are software systems designed to support users by automating specific processes, tasks, or activities. When these systems implement a conversational component to interact with users, they are also known as conversational agents or chatbots . Bots—particularly in their conversation-oriented version and AI-powered—have seen increased adoption over time for software development and engineering purposes. Despite their exciting potential, which has been further enhanced by the advent of Generative AI and Large Language Models, bots still face challenges in terms of development and integration into the development cycle, as practitioners report that bots can add difficulties rather than provide improvements. In this work, we aim to provide a taxonomy for characterizing bots, as well as a series of challenges for their adoption in software engineering, accompanied by potential mitigation strategies. To achieve our objectives, we conducted a multivocal literature review , examining both research and practitioner literature. Through such an approach, we hope to contribute to both researchers and practitioners by providing (i) a series of future research directions to pursue, (ii) a list of strategies to adopt for improving the use of bots for software engineering purposes, and (iii) fostering technology and knowledge transfer from the research field to practice—one of the primary goals of multivocal literature reviews.

Problem

• Characterize bots and conversational agents in software engineering.
• Identify challenges and mitigation strategies for bot adoption.
• Provide future research routes and practical strategies for bots.

Innovation

• Developed taxonomy for bot characterization
• Identified challenges in bot adoption
• Proposed mitigation strategies for bot integration