Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving

2024-06-19

arXiv.org

AI Summary

In LLM serving, unpredictable request generation lengths hinder efficient scheduling, limit throughput, and cause load imbalance across GPU instances. Existing sequence-level scheduling (SLS) suffers from high latency for short requests due to static batching and FCFS policies; iteration-level scheduling (ILS), though enabling dynamic batching, remains constrained by out-of-memory (OOM) risks and fixed concurrency, failing to jointly optimize throughput and load balancing. This paper proposes slice-level scheduling (SCLS), which partitions the maximum generation length into fixed-duration, memory-predictable scheduling units. SCLS introduces the first time–memory joint modeling at the slice level, integrating dynamic batching with GPU memory-aware offloading. It enables fine-grained, request-level scheduling and cross-instance load balancing while guaranteeing zero OOM. Experiments show SCLS improves throughput by up to 315.8% over SLS and ILS, significantly mitigating load skew.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation length is generally unpredictable, it is difficult to estimate the time and memory required to process requests, thus posing a challenge for effective request scheduling. Conventional sequence-level scheduling (SLS) serves requests in a first-come first-served (FCFS) manner with static batching where requests with short generation lengths are delayed until those with long ones have finished generation. Besides, to avoid out-of-memory (OOM) errors, SLS batches requests using a small batch size, which limits throughput. Recently proposed iteration-level scheduling (ILS) improves this with continuous batching, timely completing requests and dynamically adding new ones, but often limits the number of parallel-processing requests to OOM errors, thus compromising throughput. Moreover, both SLS and ILS fail to effectively balance workload across multiple LLM instances. To tackle these challenges, we propose slice-level scheduling (SCLS). By splitting the predefined maximal generation length limit into slices and serving batches slice by slice, it provides a precise range of serving time and memory usage for batched requests, laying the foundation for effective scheduling. Experiments confirm that compared with SLS and ILS schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate load imbalance with proposed batching and offloading algorithms.

Problem

• Unpredictable request generation length complicates scheduling.
• Conventional scheduling methods limit throughput and cause load imbalance.
• Proposed slice-level scheduling improves throughput and balances workload.

Innovation

• Slice-level scheduling for precise resource allocation
• Dynamic batching to enhance throughput and balance load
• Algorithm for mitigating load imbalance across instances