NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

2026-07-29

AI Summary

This work addresses the throughput and latency bottlenecks of conventional GPU architectures when handling mixed-length LLM inference requests with highly variable context lengths. The authors propose the first heterogeneous system integrating real near-memory computing (PNM) with GPUs via CXL interconnects, co-designing sparse attention execution on PNM, decoding kernels on GPUs, and a host-based scheduling runtime. They introduce a request-length-aware placement strategy and a dynamic migration mechanism that avoids recomputation, enabling efficient cross-device memory management with RPC/RDMA support. Under mixed-length workloads, the system achieves up to 5.5× higher decoding throughput and reduces P99 latency by up to 15× compared to GPU-only baselines.

Abstract

Modern LLMs and their agentic applications are broadening the range of serving workloads, spanning context lengths from a few hundred tokens to hundreds of thousands. As these requests frequently interleave within the same serving window, LLM serving systems must handle highly heterogeneous mixed-length workloads. Such mixed-length workloads expose fundamental inefficiencies in GPU-centric serving architectures, whose throughput depends on large, memory-constrained batches. In this paper, we present NELSSA, an LLM serving system that integrates GPUs with real-world Processing-near-Memory (PNM) accelerator devices to efficiently support mixed-length workloads. NELSSA employs length-based request placement to route short-context requests to GPUs and long-context requests to the PNM tier, incorporating runtime migration to accommodate dynamic context growth without recomputation. We prototype NELSSA as an end-to-end system, implementing device-level sparse attention on PNM, GPU decode kernels, and a host-side runtime that orchestrates scheduling and cross-tier memory movement over a CXL-enabled infrastructure with RPC and RDMA support. Across mixed-length LLM workloads, NELSSA improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines. Our end-to-end prototype and experimental results suggest that integrated GPU-PNM serving, enabled by CXL-based disaggregation, is a promising system paradigm for scalable and flexible LLM infrastructures that support evolving workloads.

Problem

• mixed-length workloads
• LLM serving
• heterogeneous requests
• context length variability
• GPU inefficiency

Innovation

• GPU-PNM heterogeneous system
• length-based request placement
• mixed-length LLM serving
• Processing-near-Memory (PNM)
• CXL-based disaggregation