rdma programming

Existing CXL offloading mechanisms lack fine-grained support for protocol-agnostic design trade-offs, hindering adaptation to heterogeneous data and computational requirements—and thus limiting end-to-end performance and resource efficiency in disaggregated memory systems. To address this, we propose Asynchronous Reflow, a novel protocol for CXL-based in-memory computing that decouples data and control flows, enabling asynchronous, lightweight pipelined communication between the host and compute-capable CXL memory (CCM). Furthermore, we design the hierarchical KAI system to jointly optimize near-memory computation and task offloading. Experimental evaluation demonstrates that our approach reduces end-to-end execution time by up to 50.4%, while decreasing average idle time on CCM and the host by 22.11× and 3.85×, respectively—significantly improving system utilization and latency.

RDMA offers low latency and high throughput but suffers from a weak memory model lacking formal foundations, hindering the construction of correct, composable multi-node concurrent objects. Method: We introduce LOCO—the first RDMA-native multi-node object library supporting formal verification—bridging shared-memory and distributed programming paradigms. To address RDMA consistency modeling challenges, we design Mowgli, a modular, declarative verification framework that decouples proof logic from the memory model and enables systematic correctness proofs leveraging locality properties. Contribution/Results: LOCO achieves performance comparable to custom RDMA systems while drastically simplifying the programming model. All core components are fully formally verified. LOCO provides a high-reliability, high-performance infrastructure foundation for data centers, HPC, and AI/ML workloads.

This work addresses system-level challenges introduced by CXL memory—including high latency, limited parallelism, and severe interference with DDR bandwidth (up to 81%)—tracing their root causes to unfair request scheduling, degraded inter-core synchronization, and contention for shared hardware resources. To mitigate these issues, we propose MIKU, a dynamic memory request regulation mechanism. MIKU introduces the first service-time-aware, cooperative priority scheduler for CXL and DDR requests, enabling fair allocation and efficiency-oriented coordination across heterogeneous memory resources. Leveraging microbenchmark-driven analysis, cycle-accurate hardware modeling, and protocol-stack co-optimization, MIKU recovers 98% of peak DDR bandwidth under heavy load, reduces CXL average latency variation by 42%, and significantly improves overall memory efficiency.

Existing formal semantics for RDMA lack support for remote synchronization mechanisms, making it difficult to verify the correctness of synchronization primitives such as locks. This work proposes RDMA^{TSO}_{RMW}, the first RDMA semantic model supporting remote read-modify-write (RMW) operations, reveals its weak atomicity properties, and constructs a composable synchronization library, RDMA^{WAIT}_{RMW}. Building on this foundation, the paper designs, implements, and formally verifies three classes of remote locks tailored to different usage scenarios. Furthermore, it introduces RDMA^{SC}_{RMW}, a stronger consistency model that is compatible with the high-performance LOCO library while preserving both composability and verifiability, thereby addressing a critical theoretical gap in the formal verification of RDMA-based synchronization primitives.

With CXL.mem hardware not yet commercially available, realistic evaluation of CXL-based memory systems remains infeasible. Method: This paper proposes a lightweight, pure-software simulator that addresses this gap by introducing a novel performance-monitoring-event (PME)-driven, epoch-level execution-time inference mechanism—requiring no source-code modification or full-system simulation—and supporting dynamic configuration of multi-tier memory hierarchies and access latencies. Contribution/Results: The simulator is compatible with mainstream x86 platforms and incurs only 4.41× average runtime overhead on real applications—significantly outperforming alternatives like Gem5. It accurately models CXL.mem’s high latency, low bandwidth, and heterogeneous memory characteristics. Already deployed to support system-level innovations such as memory scheduling, the simulator provides an efficient, scalable, and practical evaluation infrastructure for early-stage CXL ecosystem exploration.

This work addresses the challenges of deploying hash tables in RDMA-based remote memory under IoT big data scenarios, where limited local memory necessitates offloading to disaggregated memory systems. The key obstacles include frequent remote accesses, ineffective concurrency control, and constrained RDMA resources. To tackle these issues, the study presents the first holistic design that jointly optimizes hash collision resolution, concurrency mechanisms, and RDMA hardware characteristics. By leveraging one-sided RDMA operations, lightweight concurrency control, and careful structuring of NIC resources—including queues, atomic operations, and memory registration—the proposed paradigm significantly improves latency, throughput, and scalability of distributed key-value systems in memory-disaggregated architectures.

This work addresses the limitations of existing disaggregated in-memory databases built on one-sided RDMA, which struggle to support sophisticated concurrency control mechanisms such as starvation-freedom and priority scheduling, while also suffering from significant network amplification due to multiple round trips induced by index operations. To overcome these challenges, we propose Lotus, a system that rethinks the conventional wisdom that two-sided RDMA is inefficient due to its reliance on remote CPU involvement. By integrating lightweight caching, efficient request batching, and coordinated use of two-sided RDMA, Lotus achieves high-performance transaction processing under constrained CPU resources at memory servers. Crucially, Lotus leverages the rich semantics of two-sided RDMA to enable complex concurrency control and transforms CPU overhead into a performance advantage. Experimental results on the YCSB benchmark show that Lotus achieves up to 8.2× higher throughput and 42.9× lower p999 tail latency compared to the state-of-the-art one-sided RDMA approach.

This work addresses the performance bottleneck caused by address dependencies in remote memory access—termed the “indirection wall”—which incurs multiple round-trip latencies and severely limits applications such as graph traversal, page table lookups, distributed locking, and disaggregated large model inference. The authors propose a compact, statically verifiable eBPF-like instruction set executable on memory-side RDMA NICs, enabling pre-registered programs to resolve indirect accesses locally at the remote node and collapse multi-hop dependencies into a single communication round. Leveraging an FPGA-based memory-side NIC architecture combined with optimized remote memory primitives, the system achieves line-rate, secure processing of complex indirect accesses. Experiments demonstrate 2.85× lower latency and 3.4× higher throughput for graph traversal, 62% reduced page table lookup latency, 2.9× lower latency for contention-free distributed locks, 2.8× higher throughput for disaggregated PagedAttention, and 1.88× lower latency for MoE expert gathering.

This work addresses the performance bottleneck in existing GPU-accelerated all-flash array (AFA) systems, which rely on CPU-centric I/O architectures that incur high interaction overhead and I/O amplification, limiting end-to-end efficiency. To overcome this, the authors propose a GPU-native remote AFA system that shifts storage functionality into SSD firmware via a GPU-centric NoR software stack (GNoR) and a decentralized AFA engine (deEngine), enabling GPUs to directly and efficiently access remote AFAs without CPU intervention. The design integrates RDMA-based NVMe-oF, leverages the GPU’s SIMT parallel execution model, and employs an atomic-operation-driven I/O orchestration mechanism. Experimental results demonstrate that the proposed system achieves 3.2× higher I/O throughput and reduces application execution time by 31.1% compared to state-of-the-art approaches.

This work addresses the dual challenges of 25–35% DRAM waste due to memory fragmentation in cloud environments and high cold-start latency during MicroVM snapshot restoration. To tackle these issues, the authors propose a hierarchical memory pooling architecture that synergistically integrates CXL and RDMA. Snapshot pages are categorized by access热度 (hotness): hot pages reside in a low-latency CXL-attached pool, while cold and zero pages are stored in an RDMA-based remote memory pool. The design includes a novel ownership coherence protocol tailored for non-cache-coherent CXL devices to ensure correctness in multi-host sharing scenarios. Key innovations include the first coordinated use of CXL and RDMA for MicroVM snapshot serving, a hotness-aware snapshot format, and a hybrid loading mechanism featuring hot-page preloading with asynchronous on-demand fetching of cold pages. Experiments demonstrate that the end-to-end restoration latency is reduced by 2.2× on average compared to Firecracker and outperforms the state-of-the-art by 1.1×.