ZOR filters: fast and smaller than fuse filters

2026-02-03

AI Summary

Existing XOR/fuse filters rely on randomized peeling and succeed only with high probability, lacking deterministic guarantees that are crucial for reliable deployment. This work proposes ZOR, the first XOR/fuse-style filter to achieve deterministic construction: it replaces probabilistic retry mechanisms with a deterministic peeling algorithm that always terminates successfully. For the small fraction of keys (<1% when arity ≥ 5) that cannot be resolved by peeling, ZOR employs a compact auxiliary structure to store them while preserving the false-positive-only semantics. ZOR approaches the information-theoretic lower bound of log₂(1/ε) in space efficiency, matches fuse in query performance, and incurs only a minor latency overhead for negative queries due to auxiliary checks.

Technology Category

Search and Optimization: Distributed SearchReasoning under Uncertainty: Other Foundations of Reasoning under UncertaintyData Mining & Knowledge Management: Intelligent Query Processing

Application Category

Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSecurity and Privacy: Large-scale security measurements

Abstract

Probabilistic membership filters support fast approximate membership queries with a controlled false-positive probability $\varepsilon$ and are widely used across storage, analytics, networking, and bioinformatics \cite{chang2008bigtable,dayan2018optimalbloom,broder2004network,harris2020improved,marchet2023scalable,chikhi2025logan,hernandez2025reindeer2}. In the static setting, state-of-the-art designs such as XOR and fuse filters achieve low overhead and very fast queries, but their peeling-based construction succeeds only with high probability, which complicates deterministic builds \cite{graf2020xor,graf2022binary,ulrich2023taxor}. We introduce \emph{ZOR filters}, a deterministic continuation of XOR/fuse filters that guarantees construction termination while preserving the same XOR-based query mechanism. ZOR replaces restart-on-failure with deterministic peeling that abandons a small fraction of keys, and restores false-positive-only semantics by storing the remainder in a compact auxiliary structure. In our experiments, the abandoned fraction drops below $1\%$ for moderate arity (e.g., $N\ge 5$), so the auxiliary handles a negligible fraction of keys. As a result, ZOR filters can achieve overhead within $1\%$ of the information-theoretic lower bound $\log_2(1/\varepsilon)$ while retaining fuse-like query performance; the additional cost is concentrated on negative queries due to the auxiliary check. Our current prototype builds several-fold slower than highly optimized fuse builders because it maintains explicit incidence information during deterministic peeling; closing this optimisation gap is an engineering target.

Problem

• probabilistic membership filters
• deterministic construction
• peeling-based construction
• false-positive probability
• XOR filters

Innovation

• ZOR filters
• deterministic peeling
• probabilistic membership
• false-positive-only
• XOR-based query