Circinus: Efficient Query Planner for Compound ML Serving

2025-04-23

AI Summary

This paper addresses real-time query planning for multi-operator AI pipelines in edge-cloud collaborative environments under concurrent Service-Level Objective (SLO) constraints on latency, accuracy, and cost. Challenges include heterogeneous edge capabilities, diverse SLO requirements, and a high-dimensional optimization space. Method: We propose the first decomposition-based planning framework supporting joint multi-query and multi-dimensional SLO optimization. It introduces an incremental pruning mechanism leveraging plan similarity and an accuracy-aware early-termination profiling method for rapid performance estimation. Results: Experiments demonstrate a 3.2–5.0× improvement in service goodput, planning latency reduced to sub-second levels (4.2–5.8× speedup), and 3.2–4.0× reduction in deployment cost—significantly outperforming single-tier baseline approaches.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rise of compound AI serving -- integrating multiple operators in a pipeline that may span edge and cloud tiers -- enables end-user applications such as autonomous driving, generative AI-powered meeting companions, and immersive gaming. Achieving high service goodput -- i.e., meeting service level objectives (SLOs) for pipeline latency, accuracy, and costs -- requires effective planning of operator placement, configuration, and resource allocation across infrastructure tiers. However, the diverse SLO requirements, varying edge capabilities, and high query volumes create an enormous planning search space, rendering current solutions fundamentally limited for real-time serving and cost-efficient deployments. This paper presents Circinus, an SLO-aware query planner for large-scale compound AI workloads. Circinus novelly decomposes multi-query planning and multi-dimensional SLO objectives while preserving global decision quality. By exploiting plan similarities within and across queries, it significantly reduces search steps. It further improves per-step efficiency with a precision-aware plan profiler that incrementally profiles and strategically applies early stopping based on imprecise estimates of plan performance. At scale, Circinus selects query-plan combinations to maximize global SLO goodput. Evaluations in real-world settings show that Circinus improves service goodput by 3.2-5.0$ imes$, accelerates query planning by 4.2-5.8$ imes$, achieving query response in seconds, while reducing deployment costs by 3.2-4.0$ imes$ over state of the arts even in their intended single-tier deployments.

Problem

• Optimizing compound AI serving pipelines across edge and cloud tiers
• Planning operator placement and resource allocation for diverse SLOs
• Reducing search space for real-time query planning efficiency

Innovation

• Decomposes multi-query planning and SLO objectives
• Reduces search steps via plan similarities
• Uses precision-aware plan profiler for efficiency