MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

2026-09-21

AI Summary

研究通过MCP-GRANITE框架探讨了工具接口粒度对基于MCP的LLM代理性能的影响，发现4工具接口在任务完成和参数准确性上表现最佳。

Abstract

As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet underexplored factor. How funψtionality is decomposed into tools affects whether an agent can select the right tool and construct valid arguments. This choice is especially consequential at the edge, where resource constraints limit which models can run locally and scaling up is often not an option. We present MCP-GRANITE, an open-source extensible benchmark framework that treats tool-interface granularity as a controlled variable for MCP-based agents, evaluated under edge and IoT scenarios. It comprises 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels from fine-grained primitive tools to a single tool. We evaluate 9 locally deployed models (268M-20.9B parameters) across 8,748 trials using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics. Results show that a 4-tool interface offers the best trade-off, improving task completion by 16.4% over fine-grained primitives and 33.6% over a single monolithic tool, while nearly doubling argument accuracy. Model size is only weakly correlated with task completion and strongly with latency, while its association with argument accuracy is less robust, and a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one. These findings identify tool-interface granularity as a key design parameter for MCP-based agents.

Problem

• LLM agents
• MCP
• tool-interface design
• granularity
• edge computing

Innovation

• MCP-GRANITE
• tool-interface granularity
• edge and IoT scenarios
• task completion
• argument accuracy