Rain: RDMA-assisted In-Network Scheduling for Microsecond-scale Workloads

2026-06-02

AI Summary

Existing in-network schedulers struggle to simultaneously achieve microsecond-scale tail latency and high throughput, primarily due to the lack of global coordination, reliance on packet re-circulation, and inadequate support for large tasks. This work proposes Rain, an RDMA-assisted scheduler leveraging programmable switches, which introduces a novel bidirectional on-chip queuing mechanism to match tasks with worker-node tokens and integrates a switch-driven RDMA engine to pre-write large tasks. Challenging the common assumption that shallow queues inherently reduce tail latency, Rain employs a runtime-adaptive strategy to dynamically optimize queue depth and slice-to-queue mappings. Experimental results demonstrate that, under identical tail latency constraints in RocksDB, Rain achieves 1.75× higher throughput compared to the state-of-the-art solution.

Abstract

Modern data center applications increasingly require microsecond-scale service time with strict tail latency requirements, which can hardly be realized with existing in-network task schedulers due to their inherent limitations. Specifically, software-based schedulers struggle to balance throughput and latency, while switch-based designs either lack global coordination, rely on packet recirculation heavily, or only offer limited support for large tasks. In light of these restrictions of the state-of-the-arts (SOTAs), we, in this work, propose Rain, an RDMA-assisted in-network scheduler built atop programmable switches that maintains centralized queues while bounding worker-local queues. Rain introduces a bidirectional on-switch queuing mechanism to buffer and match tasks and worker-issued tokens directly in the switch, avoiding worker-side polling and approximating the optimal behavior of join-bounded-shortest-queue without global aggregation. A switch-driven RDMA engine pre-writes arbitrarily large tasks via one-sided WRITE multicasts, keeping only compact metadata on the switch. Slice-aware scheduling further localizes decisions to more homogeneous queues, reducing dispersion-induced head-of-line blocking. Moreover, our study reveals that real-world systems can diverge from theoretical predictions: shallower worker queues do not always improve tail latency. Leveraging this insight, Rain incorporates an adaptive scheduling strategy to optimize worker queue depths and worker-to-slice mappings at runtime. Evaluations with the real-world application RocksDB show that Rain achieves 1.75x higher throughput than the best-performing SOTA while satisfying the same tail latency requirement.

Problem

• microsecond-scale workloads
• tail latency
• in-network scheduling
• RDMA
• programmable switches

Innovation

• RDMA-assisted scheduling
• in-network computing
• programmable switches
• tail latency optimization
• adaptive queue management