Collaboration Dynamics and Reliability Challenges of Multi-Agent LLM Systems in Finite Element Analysis

2024-08-23

AI Summary

This work addresses the degradation of reasoning quality and unreliable verification in LLM-based multi-agent systems for scientific computing—specifically linear-elastic finite element analysis—caused by collaborative dynamics. We systematically identify three systemic failure modes: confirmation bias, premature consensus, and verification–validation decoupling, leading to undetected physics-inconsistent code. Building upon the AutoGen framework, we design a role-specialized tri-agent system (Coder/Executor/Critic) and evaluate it via controlled dialogue experiments under a dual-criteria assessment paradigm: physical consistency and executable correctness. Results show that functional complementarity outweighs team size; Critic involvement achieves 100% correctness in both physics and visualization; and confirmation bias is detected with 85–92% accuracy. Based on these findings, we propose three actionable design principles—role differentiation, multi-level verification, and anti-premature-convergence interaction—to establish a foundation for engineering-grade trustworthy multi-agent systems.

Technology Category

• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Intelligent Robots: Multi-Robot Systems

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large Language Model (LLM)-based multi-agent systems are increasingly applied to automate computational workflows in science and engineering. However, how inter-agent dynamics influence reasoning quality and verification reliability remains unclear. We study these mechanisms using an AutoGen-based multi-agent framework for linear-elastic Finite Element Analysis (FEA), evaluating seven role configurations across four tasks under a fixed 12-turn conversation limit. From 1,120 controlled trials, we find that collaboration effectiveness depends more on functional complementarity than team size: the three-agent Coder-Executor-Critic configuration uniquely produced physically and visually correct solutions, while adding redundant reviewers reduced success rates. Yet three systematic failure modes persist: (1) affirmation bias, where the Rebuttal agent endorsed rather than challenged outputs (85-92% agreement, including errors); (2) premature consensus caused by redundant reviewers; and (3) a verification-validation gap where executable but physically incorrect code passed undetected. No agent combination successfully validated constitutive relations in complex tasks. Building on theories of functional diversity, role differentiation, and computational validation, we propose actionable design principles: (i) assign complementary agent roles, (ii) enforce multi-level validation (execution, specification, physics), and (iii) prevent early consensus through adversarial or trigger-based interaction control. These findings establish a principled foundation for designing trustworthy LLM collaborations in engineering workflows.

Problem

• Examining how inter-agent dynamics affect reasoning quality in multi-agent LLM systems
• Addressing systematic failure modes in automated computational workflows for engineering
• Developing design principles for reliable multi-agent collaboration in finite element analysis

Innovation

• Complementary agent roles enhance collaboration effectiveness
• Multi-level validation prevents executable but incorrect outputs
• Adversarial interactions prevent premature consensus in teams