MCPGen: Benchmarking LLMs on Executable MCPWorkflow Development

2026-09-20

AI Summary

研究通过MCPGen基准测试评估大语言模型生成可执行工作流的能力，涵盖结构、工具实现、模式绑定和运行时连接的一致性问题。

Abstract

We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce \textbf{MCPGen}, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5\% on workflow reconstruction, but no model exceeds 57\% end-to-end execution success. Per-tool unit-test pass rates reach 63.8\%, while project-level integration success does not exceed 45\%, suggesting that integration remains a major bottleneck even when isolated tool tests pass.

Problem

• LLMs
• Executable Workflows
• Cross-layer Consistency
• End-to-end Execution
• Integration Bottleneck

Innovation

• MCPGen
• executable benchmark
• cross-layer consistency
• end-to-end execution
• workflow development