{epsilon}-Cost Sharding: Scaling Hypergraph-Based Static Functions and Filters to Trillions of Keys

2025-03-24

AI Summary

To address scalability bottlenecks of static functions and filters in trillion-scale key-value systems, this paper proposes HyperFunc—a hypergraph-based framework for constructing verifiable static functions (VFunc) and filters (VFilter). The core innovation is the ε-cost partitioning mechanism: achieving zero metadata overhead and eliminating computational chain dependencies by unifying fused graph partitioning (for large partitions) with lazy Gaussian elimination (for small partitions), thereby approaching the information-theoretic lower bound. Integrated with minimal-perfect hashing via monotone minimal perfect hash functions (MWHC) and static hash construction, HyperFunc incurs only 10.5% space overhead above the theoretical lower bound and adds merely a few nanoseconds to query latency. On commodity hardware, offline construction attains 60 ns per key, enabling efficient deployment of ultra-large-scale static functions and filters.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Privacy-enhancing technologies

Abstract

We describe a simple and yet very scalable implementation of static functions (VFunc) and of static filters (VFilter) based on hypergraphs. We introduce the idea of {epsilon}-cost sharding, which allows us to build structures that can manage trillions of keys, at the same time increasing memory locality in hypergraph-based constructions. Contrarily to the commonly used HEM sharding method, {epsilon}-cost sharding does not require to store of additional information, and does not introduce dependencies in the computation chain; its only cost is that of few arithmetical instructions, and of a relative increase {epsilon} in space usage. We apply {epsilon}-cost sharding to the classical MWHC construction, but we obtain the best result by combining Dietzfelbinger and Walzer's fuse graphs for large shards with lazy Gaussian elimination for small shards. We obtain large structures with an overhead of 10.5% with respect to the information-theoretical lower bound and with a query time that is a few nanoseconds away from the query time of the non-sharded version, which is the fastest currently available within the same space bounds. Besides comparing our structures with a non-sharded version, we contrast its tradeoffs with bumped ribbon constructions, a space-saving alternative to hypergraph-based static functions and filters, which provide optimum space consumption but slow construction and query time (though construction can be parallelized very efficiently). We build offline a trillion-key filter using commodity hardware in just 60 ns/key.

Problem

• Scaling hypergraph-based static functions to trillions of keys
• Improving memory locality in hypergraph constructions
• Reducing space and computation costs in sharding methods

Innovation

• Uses ε-cost sharding for hypergraph scalability
• Combines fuse graphs with lazy Gaussian elimination
• Achieves 10.5% overhead near theoretical limit