Two-sided RDMA Striking Back for Disaggregated Memory Databaases

2026-07-28

AI Summary

This work addresses the limitations of existing disaggregated in-memory databases built on one-sided RDMA, which struggle to support sophisticated concurrency control mechanisms such as starvation-freedom and priority scheduling, while also suffering from significant network amplification due to multiple round trips induced by index operations. To overcome these challenges, we propose Lotus, a system that rethinks the conventional wisdom that two-sided RDMA is inefficient due to its reliance on remote CPU involvement. By integrating lightweight caching, efficient request batching, and coordinated use of two-sided RDMA, Lotus achieves high-performance transaction processing under constrained CPU resources at memory servers. Crucially, Lotus leverages the rich semantics of two-sided RDMA to enable complex concurrency control and transforms CPU overhead into a performance advantage. Experimental results on the YCSB benchmark show that Lotus achieves up to 8.2× higher throughput and 42.9× lower p999 tail latency compared to the state-of-the-art one-sided RDMA approach.

Abstract

RDMA has enabled high-speed data access and low-latency communication in disaggregated memory databases. While various optimization techniques have been proposed to accelerate transactions with RDMA in this setting, two-sided RDMA has been largely underexplored in favor of one-sided RDMA due to its remote CPU involvement. However, the heavy use of one-sided RDMA introduces fundamental limitations. Its limited APIs cannot express complex system functions such as starvation prevention, priority-based scheduling, and preemption, which are all critical functions in concurrency control protocols. Moreover, indexing requires multiple network round-trips, causing network amplification. In this work, we revisit the long-standing debate between one-sided RDMA and two-sided RDMA in the context of disaggregated memory databases. We present Lotus, which addresses the conventional limitation of two-sided RDMA, i.e., CPU bottlenecks in memory servers, by leveraging the rich functionality of two-sided RDMA with two key optimization techniques: (1) lightweight caching and (2) efficient batching. Lotus demonstrates that limited CPU resources in memory servers, when intelligently utilized, can transform a perceived weakness into a significant advantage. Our experimental study shows that Lotus achieves up to 8.2$\times$ higher throughput and 42.9$\times$ lower p999 tail latency than state-of-the-art one-sided RDMA-based approaches in YCSB benchmark.

Problem

• disaggregated memory databases
• one-sided RDMA
• two-sided RDMA
• concurrency control
• network amplification

Innovation

• two-sided RDMA
• disaggregated memory databases
• concurrency control
• lightweight caching
• efficient batching