DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

2026-07-30

AI Summary

This work addresses the underutilization of GPU resources in large language model inference systems, which are often over-provisioned to handle peak loads, leaving spare capacity idle during off-peak periods. To this end, we propose DeltaServe, a host-agnostic co-serving architecture that dynamically repurposes idle compute for LoRA fine-tuning tasks while strictly meeting inference service-level objectives (SLOs). DeltaServe’s key innovation lies in sharing the execution structure between inference prefilling and LoRA forward passes, coupled with an SLO-aware scheduler that enables efficient co-location without additional hardware overhead. The system integrates lightweight hooks supporting multi-LoRA batching and CUDA Graph–based latency modeling, offering compatibility with mainstream serving engines such as vLLM and SGLang. Experiments on real-world production workloads show that DeltaServe achieves 2.9× higher fine-tuning throughput than LLMStation and 39% improvement over a vLLM+torchtune baseline, all while maintaining 100% SLO compliance for inference requests.

Abstract

LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.

Problem

• LLM serving
• resource underutilization
• LoRA fine-tuning
• inference SLO
• co-serving

Innovation

• co-serving
• LoRA fine-tuning
• SLO-aware scheduling
• host-agnostic
• CUDA-graph-aware latency model