MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models

2025-07-17

AI Summary

Current LLM agent evaluation relies on static benchmarks and manual annotation, suffering from poor scalability and low efficiency. To address this, we propose the first open-source, automated, deep-evaluation framework for LLM agents based on the Model Context Protocol (MCP). Our method integrates MCP-compliant tool orchestration, adaptive task modeling, and a standardized multi-dimensional metric suite to enable fully automated, reproducible, end-to-end assessment across diverse domains—without human intervention. The framework dynamically generates realistic, domain-specific tasks, natively incorporates external tools, and supports fine-grained performance analysis along dimensions including correctness, efficiency, robustness, and tool utilization. We validate it across five real-world scenarios, demonstrating substantial improvements in both evaluation throughput and analytical depth. The framework is publicly released as an open-source project, establishing a standardized, scalable infrastructure for rigorous, automated AI agent evaluation.

Technology Category

Multiagent Systems: Modeling other AgentsCognitive Modeling & Cognitive Systems: Agent ArchitecturesMachine Learning: Large Multimodal Models (LMMs)

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The rapid rise of Large Language Models (LLMs)-based intelligent agents underscores the need for robust, scalable evaluation frameworks. Existing methods rely on static benchmarks and labor-intensive data collection, limiting practical assessment. We introduce oursystemname, an open-source Model Context Protocol (MCP)-based framework that automates end-to-end task generation and deep evaluation of LLM agents across diverse domains. MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines. Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance. We publicly release MCPEval https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and standardized LLM agent evaluation.

Problem

• Automates end-to-end task generation for LLM agents
• Standardizes metrics for diverse domain evaluations
• Eliminates manual effort in evaluation pipelines

Innovation

• Automates end-to-end task generation and evaluation
• Standardizes metrics and integrates with agent tools
• Eliminates manual effort in evaluation pipelines