Scalable LLM Agent Tool Access in the Cloud

2026-07-16

AI Summary

This work addresses key scalability challenges faced by large language model (LLM) agents operating in cloud environments under the Model Context Protocol (MCP), including difficulties in tool integration, limited context windows, high inference overhead, and complex session affinity. To overcome these issues, the paper introduces the first cloud-native gateway architecture specifically designed for MCP. The proposed system decouples direct dependencies between agents and tools through a protocol adaptation layer, a hybrid retrieval mechanism, session-aware routing, and dynamic tool recommendation, enabling unified access to and efficient orchestration of heterogeneous services. Evaluated at scale, the system supports over 3,000 tools, achieves a Top-15 recall rate of 98%, reduces tool selection latency by 8.9×, and decreases token consumption by 23.8×, while maintaining low latency and high invocation success rates under horizontal scaling.

Abstract

LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.

Problem

• LLM agents
• Model Context Protocol
• cloud-scale
• tool calling
• session affinity

Innovation

• Model Context Protocol
• cloud-scale gateway
• tool retrieval
• session-aware routing
• LLM agent