Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

2026-07-25

AI Summary

This work addresses the performance bottlenecks in cross-device GPU sharing within local area networks, which are primarily constrained by limited network bandwidth, frequent CUDA invocations, and contention among concurrent tasks. To overcome these challenges, the authors propose an adaptive and efficient GPU sharing framework that enables resource sharing across heterogeneous devices through remote CUDA API calls. The framework innovatively integrates automatic model weight caching, an asynchronous API execution engine, dynamic task scheduling based on real-time network and GPU load conditions, and a distributed CUDA context consistency mechanism. Experimental results demonstrate that, across diverse NVIDIA GPUs and AI workloads, the proposed approach achieves 1.4–24.2× higher API invocation efficiency and up to 1.79× improvement in system throughput, significantly outperforming existing solutions.

Abstract

This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency and up to 1.79 times higher system throughput.

Problem

• GPU sharing
• CUDA API remoting
• network bottleneck
• LAN
• AI inference

Innovation

• CUDA API remoting
• GPU sharing
• network-efficient
• asynchronous execution
• runtime scheduling