Explicit Min-wise Hash Families with Optimal Size

2025-10-11

AI Summary

This work addresses the explicit construction of min-wise hash families and their k-min-wise generalizations, aiming for subconstant error with optimal randomness complexity O(k log N). Prior constructions failed to simultaneously achieve low error, explicitness, and polynomial size. To overcome this, we introduce a novel pseudorandom generator based on combinatorial rectangles and single-read branching programs, and refine the Nisan–Zuckerman framework. Our construction yields the first explicit k-min-wise family that uniformly selects the k smallest elements from any subset—a property previously unattained. When k = log^{O(1)} N, it achieves error 2^{-Ω(log N / log log N)}, outperforming all prior O(k log N)-bit schemes. This provides theoretically optimal, efficient tools for ℓ₀ sampling, data summarization, and web duplicate detection.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Other Foundations of Machine Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study explicit constructions of min-wise hash families and their extension to $k$-min-wise hash families. Informally, a min-wise hash family guarantees that for any fixed subset $Xsubseteq[N]$, every element in $X$ has an equal chance to have the smallest value among all elements in $X$; a $k$-min-wise hash family guarantees this for every subset of size $k$ in $X$. Min-wise hash is widely used in many areas of computer science such as sketching, web page detection, and $ell_0$ sampling. The classical works by Indyk and Pătraşcu and Thorup have shown $Θ(log(1/δ))$-wise independent families give min-wise hash of multiplicative (relative) error $δ$, resulting in a construction with $Θ(log(1/δ)log N)$ random bits. Based on a reduction from pseudorandom generators for combinatorial rectangles by Saks, Srinivasan, Zhou and Zuckerman, Gopolan and Yehudayoff improved the number of bits to $O(log Nloglog N)$ for polynomially small errors $δ$. However, no construction with $O(log N)$ bits (polynomial size family) and sub-constant error was known before. In this work, we continue and extend the study of constructing ($k$-)min-wise hash families from pseudorandomness for combinatorial rectangles and read-once branching programs. Our main result gives the first explicit min-wise hash families that use an optimal (up to constant) number of random bits and achieve a sub-constant (in fact, almost polynomially small) error, specifically, an explicit family of $k$-min-wise hash with $O(klog N)$ bits and $2^{-O(log N/loglog N)}$ error. This improves all previous results for any $k=log^{O(1)}N$ under $O(k log N)$ bits. Our main techniques involve several new ideas to adapt the classical Nisan-Zuckerman pseudorandom generator to fool min-wise hashing with a multiplicative error.

Problem

• Constructing explicit min-wise hash families with optimal size
• Achieving sub-constant error using minimal random bits
• Extending k-min-wise hashing with improved pseudorandom generators

Innovation

• Uses pseudorandom generators for combinatorial rectangles
• Adapts Nisan-Zuckerman generator for min-wise hashing
• Achieves optimal random bits with sub-constant error