Kalypso: Relational LLM Serving

2026-07-26

AI Summary

This work addresses the inefficiency of existing LLM serving systems, which are request-centric and lack awareness of semantic query plans, thereby hindering inter-operator optimization. To overcome this limitation, we propose a relational LLM serving abstraction that, for the first time, integrates query plan awareness into LLM inference. By leveraging structure-aware pipelined execution and cross-operator reuse of KV caches, our approach significantly improves efficiency while preserving semantic correctness and generation accuracy. We further design an adaptive, memory-aware scheduling algorithm that dynamically balances upstream parallelism, downstream progress, and GPU memory pressure. Experimental results demonstrate that our method achieves up to a 4.57× speedup across diverse workloads, substantially enhancing the end-to-end execution efficiency of semantic queries.

Abstract

Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan, leaving substantial performance opportunities unused. This paper introduces relational LLM serving, an abstraction that makes LLM serving aware of semantic query structure while preserving query semantics and output accuracy. The key opportunity is pipelined execution across semantic operators: when intermediate tuples flow directly from one operator to the next, their KV-cache state can be reused instead of recomputed. We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm. Kalypso addresses a new online scheduling problem in which pipelined operator execution is coupled with GPU memory pressure management to reuse KV-cache state in the serving engine before eviction. Its scheduler continuously adjusts memory allocations to balance upstream parallelism, downstream progress, and GPU utilization. Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads, demonstrating that query-aware LLM serving can substantially improve the efficiency of semantic query execution.

Problem

• LLM serving
• semantic query processing
• KV-cache reuse
• query-aware scheduling
• GPU memory management

Innovation

• relational LLM serving
• KV-cache reuse
• pipelined execution
• memory-aware scheduling
• semantic query processing