Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management

2025-09-16

AI Summary

To address long queueing delays, high hardware costs, and low cluster utilization caused by static resource allocation in LLM inference, this paper proposes a dynamic scheduling framework tailored for throughput-oriented workloads on heterogeneous, ephemeral GPU clusters. The core contribution is a general-purpose context management mechanism that enables seamless migration and reuse of LLM inference states across unstable resources. It achieves this through context partitioning, caching, cross-node state recovery, and consistency maintenance. This mechanism supports fine-grained, elastic resource scaling without pre-emptive allocation. Experimental evaluation on ephemeral GPU clusters demonstrates that our approach reduces task execution time by 98.1%, while significantly improving resource utilization and inference throughput.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The widespread growth in LLM developments increasingly demands more computational power from clusters than what they can supply. Traditional LLM applications inherently require huge static resource allocations, which force users to either wait in a long job queue and accept progress delay, or buy expensive hardware to fulfill their needs and exacerbate the demand-supply problem. However, not all LLM applications are latency-sensitive and can instead be executed in a throughput-oriented way. This throughput orientation allows a dynamic allocation that opportunistically pools available resources over time, avoiding both the long queue and expensive GPU purchases. Effectively utilizing opportunistic resources brings numerous challenges nevertheless. Our solution, pervasive context management, exploits the common computational context in LLM applications and provides mechanisms and policies that allow seamless context reuse on opportunistic resources. Our evaluation shows an LLM application with pervasive context management on opportunistic resources reduces its execution time by 98.1%.

Problem

• Optimizing throughput-oriented LLM inference on heterogeneous GPU clusters
• Reducing execution delays and expensive hardware requirements
• Enabling dynamic resource allocation through context reuse

Innovation

• Dynamic resource allocation on opportunistic GPU clusters
• Pervasive context management for computational reuse
• Throughput-oriented execution reducing latency constraints