Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context

2025-09-18

AI Summary

Large language model (LLM) agents in multi-agent expert systems suffer from cognitive failures—including bias expression, information drift, and erroneous tool invocation—due to dynamic environmental shifts and misalignment with static evaluation protocols. Method: We propose a dynamic evaluation and context optimization framework featuring a novel ternary diagnostic system that integrates gold-standard annotations, behaviorally mutated silver datasets, and LLM-based agent judges. This system constructs a vectorized improvement suggestion graph for precise cognitive error localization, interpretable root-cause attribution, and cross-instance correction. Contributions/Results: (i) A controlled behavioral-mutation-driven dynamic evaluation protocol; (ii) fine-grained, reproducible performance scoring via LLM agent judges; and (iii) a context repair mechanism grounded in expert behavior transfer. Evaluated on a recruitment assistant system, our approach significantly improves reasoning accuracy and stylistic consistency.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid evolution of neural architectures - from multilayer perceptrons to large-scale Transformer-based models - has enabled language models (LLMs) to exhibit emergent agentic behaviours when equipped with memory, planning, and external tool use. However, their inherent stochasticity and multi-step decision processes render classical evaluation methods inadequate for diagnosing agentic performance. This work introduces a diagnostic framework for expert systems that not only evaluates but also facilitates the transfer of expert behaviour into LLM-powered agents. The framework integrates (i) curated golden datasets of expert annotations, (ii) silver datasets generated through controlled behavioural mutation, and (iii) an LLM-based Agent Judge that scores and prescribes targeted improvements. These prescriptions are embedded into a vectorized recommendation map, allowing expert interventions to propagate as reusable improvement trajectories across multiple system instances. We demonstrate the framework on a multi-agent recruiter-assistant system, showing that it uncovers latent cognitive failures - such as biased phrasing, extraction drift, and tool misrouting - while simultaneously steering agents toward expert-level reasoning and style. The results establish a foundation for standardized, reproducible expert behaviour transfer in stochastic, tool-augmented LLM agents, moving beyond static evaluation to active expert system refinement.

Problem

• Diagnosing cognitive failures in multi-agent expert systems
• Evaluating agentic performance beyond classical methods
• Transferring expert behavior into LLM-powered agents

Innovation

• Dynamic evaluation protocols diagnose cognitive failures
• Mutation processing context enables expert behavior transfer
• Vectorized recommendation map propagates reusable improvement trajectories