PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving

2026-08-05

AI Summary

This work addresses the memory bandwidth bottleneck and high cost that hinder multi-LoRA serving, where conventional approaches relying on CPU DRAM and PCIe transfers suffer from high latency and poor scalability. The paper introduces the first integration of near-data processing (NDP) with CXL/NVLink pooled memory, co-locating LoRA adapters and KV caches in a shared memory pool. GPUs directly drive computation using native memory instructions and return only compact results, while a link-parameterized cost model dynamically selects execution strategies. This hardware-aware approach enables efficient adapter scheduling and cache management, achieving a 6.6× average reduction in decoding latency when supporting 1,000 adapters on a single H100 GPU, with only a 3.4% increase in chip area. Moreover, it saturates throughput at just 32 GB/s bandwidth, significantly enhancing energy efficiency and cross-interconnect scalability.

Abstract

Multi-LoRA serving is how one base model becomes thousands of specialized variants, one adapter per user, task, or agent, and the deployments can hold 1000-plus adapters. Serving them is hard because the workload inverts what GPUs provide: terabytes of memory against only tens of TFLOPS, and because every published system stages its adapters from CPU DRAM over PCIe, where each access pays a kernel stop and a host-run copy and capacity ends at the motherboard's DIMM slots. Meanwhile, memory-semantic fabrics such as CXL and NVLink are converging on pooled memory that an accelerator addresses with its own loads and stores, and near-data processing (NDP) can place compute beside the pooled data. How to serve multi-LoRA workloads on such hardware remains unexplored. This paper introduces PLoRA, an NDP-enhanced pooled-memory system for cost-efficient multi-LoRA serving. PLoRA keeps adapters and KV cache in the pool and returns only reduced results over the link, through a read-compute interface the GPU drives with its own loads and stores. Above this architecture, a GPU memory management system picks among four LoRA and two attention execution strategies for each adapter and caches the most performance-critical bytes in GPU memory, guided by a link-parameterized cost model. On one H100 serving 1000 adapters, PLoRA attains the lowest decode latency on every model and workload we measure, averaging 6.6x below a real-machine S-LoRA at under 3.4% added device area. The link itself stops mattering: throughput saturates at 32 GB/s on short contexts, a quarter of CXL 3.1, and the verdict survives scale: per-GPU demand falls from 7B to a modeled 1.2T deployment once adapter traffic shards with the tensor parallelism. The design runs unchanged from CXL-class to NVLink-class fabrics, and surplus bandwidth buys pooled capacity rather than speed.

Problem

• Multi-LoRA serving
• pooled memory
• near-data processing
• memory-semantic fabrics
• adapter management

Innovation

• Near-Data Processing
• Pooled Memory
• Multi-LoRA Serving
• Memory-Semantic Interconnect
• Cost-Efficient Inference