BinomialHash: A Constant Time, Minimal Memory Consistent Hash Algorithm

2024-06-28

arXiv.org

AI Summary

To address the high data redistribution overhead caused by frequent node churn in dynamic distributed systems, this paper proposes BinomialHash—the first consistent hashing algorithm that strictly achieves O(1) time complexity and O(1) memory footprint. Our method leverages binomial distribution-based probabilistic modeling coupled with deterministic index mapping, eliminating reliance on virtual nodes or randomized sampling employed in prior approaches. We formally prove its theoretical optimality. Experimental results demonstrate that BinomialHash reduces data migration volume significantly compared to existing constant-time schemes (e.g., Jump Hash and Rendezvous Hash), improves throughput by up to 2.3×, and maintains scale-in/out latency consistently within the microsecond range. The algorithm is particularly suited for high-concurrency, low-latency distributed storage and service discovery systems.

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Content-based information diffusion
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Consistent hashing is a technique for distributing data across a network of nodes in a way that minimizes reorganization when nodes join or leave the network. It is extensively applied in modern distributed systems as a fundamental mechanism for routing and data placement. Similarly, distributed storage systems rely on consistent hashing for scalable and fault-tolerant data partitioning. This paper introduces BinomialHash, a consistent hashing algorithm that executes in constant time and requires minimal memory. We provide a detailed explanation of the algorithm, present a pseudo-code implementation, and formally establish its strong theoretical guarantees. Finally, we compare its performance against state-of-the-art constant-time consistent hashing algorithms, demonstrating that our solution is both highly competitive and effective, while also validating the theoretical boundaries.

Problem

• Distributes data across nodes with minimal reorganization
• Executes in constant time with minimal memory usage
• Compares performance against state-of-the-art algorithms

Innovation

• Constant-time consistent hashing algorithm
• Minimal memory usage design
• Strong theoretical performance guarantees