Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering

2025-05-07

AI Summary

To address the lack of trustworthiness and accountability in human-AI task allocation within LLM-driven multi-agent software engineering systems, this paper proposes the first RACI-based human-AI collaboration framework, embedding core trustworthy AI principles throughout task assignment, responsibility delineation, and risk management. Methodologically, it integrates RACI responsibility matrix modeling, an LLM-powered multi-agent architecture, and techniques for aligning system behavior with trustworthy AI governance standards. The contributions include: (1) a deployable framework design grounded in RACI semantics; (2) comprehensive implementation guidelines; and (3) a functional prototype system. Empirical evaluation demonstrates significant improvements in explainability, traceability, and accountability of human-AI collaborative software development. The framework advances both theoretical foundations and practical paradigms for trustworthy autonomous systems, offering a principled approach to governing AI-assisted engineering workflows.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Teamwork

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Multi-agent autonomous systems (MAS) are better at addressing challenges that spans across multiple domains than singular autonomous agents. This holds true within the field of software engineering (SE) as well. The state-of-the-art research on MAS within SE focuses on integrating LLMs at the core of autonomous agents to create LLM-based multi-agent autonomous (LMA) systems. However, the introduction of LMA systems into SE brings a plethora of challenges. One of the major challenges is the strategic allocation of tasks between humans and the LMA system in a trustworthy manner. To address this challenge, a RACI-based framework is proposed in this work in progress article, along with implementation guidelines and an example implementation of the framework. The proposed framework can facilitate efficient collaboration, ensure accountability, and mitigate potential risks associated with LLM-driven automation while aligning with the Trustworthy AI guidelines. The future steps for this work delineating the planned empirical validation method are also presented.

Problem

• Strategic task allocation between humans and LLM-based multi-agent systems
• Ensuring trustworthy collaboration in human-agent software engineering
• Mitigating risks in LLM-driven automation while maintaining accountability

Innovation

• RACI-based framework for human-agent task allocation
• Ensures accountability in LLM-driven automation
• Aligns with Trustworthy AI guidelines