🤖 AI Summary
This study addresses the challenge of simultaneously achieving low latency, high throughput, and cost efficiency in enterprise-scale composite AI systems under concurrent heterogeneous model invocations. It presents the first systematic analysis of system-specific issues, including multi-model fan-out overhead, cascading cold-start propagation, and dynamic heterogeneity in scaling behavior. To tackle these challenges, the authors propose a modular, platform-agnostic inference architecture that integrates serverless computing, dynamic autoscaling, MLOps pipelines, and cooperative multi-model scheduling. Experimental results demonstrate that the proposed approach reduces P95 tail latency by over 50%, increases throughput by up to 3.9×, and achieves 30%–40% cost savings compared to baseline systems.
📝 Abstract
Modern enterprise AI applications increasingly rely on compound AI systems - architectures that compose multiple models, retrievers, and tools to accomplish complex tasks. Deploying such systems in production demands inference infrastructure that can efficiently serve concurrent, heterogeneous model invocations while maintaining cost-effectiveness and low latency. This paper presents a production deployment study of a modular, platform-agnostic inference architecture developed at Salesforce to support compound AI use cases including Agentforce (autonomous AI agents) and ApexGuru (AI-powered code analysis). The system integrates serverless execution, dynamic autoscaling, and MLOps pipelines to deliver consistent low-latency inference across multi-component agent workflows. We report production results demonstrating over 50% reduction in tail latency (P95), up to 3.9x throughput improvement, and 30 to 40% cost savings compared to prior static deployments. We further present a novel analysis of compound-system-specific challenges including multi-model fan-out overhead, cascading cold-start propagation, and heterogeneous scaling dynamics that emerge uniquely when serving agentic workloads. Through detailed case studies and operational lessons, we illustrate how the architecture enables compound AI systems to scale model invocations in parallel, handle bursty multi-agent workloads, and support rapid model iteration - capabilities essential for operationalizing agentic AI at enterprise scale.