EMA: Elastic and Performance Transparent Memory Across GPUs

2026-09-22

AI Summary

EMA系统通过允许GPU间动态共享内存解决多GPU服务器中内存利用不均问题，确保性能透明且提高吞吐量。

Abstract

Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2X capacity, and maintains latency similar to the static local baseline.

Problem

• multi-GPU servers
• dynamic memory demands
• resource sharing
• elastic resource model
• memory utilization

Innovation

• Elastic Memory
• Performance Transparency
• Memory Sharing Across GPUs
• Prefetching