🤖 AI Summary
This paper studies the fair top-$k$ selection problem: ensuring representative inclusion of minority or historically disadvantaged groups when selecting the top $k$ items from high-dimensional data using a linear scoring function. We first establish the inherent computational hardness of this problem in high dimensions by proving its NP-hardness. For small $k$, we propose an efficient algorithm with theoretical guarantees and its parallel implementation; for large $k$, we design a scalable, practical surrogate method. Our approach integrates linear weighted modeling, rigorous complexity analysis, and hardware-aware optimization targeting multi-core CPUs and GPUs. Extensive experiments on real-world datasets demonstrate that our methods achieve speedups of several orders of magnitude over state-of-the-art baselines, significantly improving both efficiency and scalability for fair top-$k$ selection in large-scale, high-dimensional settings.
📝 Abstract
Selecting a subset of the $k$"best"items from a dataset of $n$ items, based on a scoring function, is a key task in decision-making. Given the widespread use of automated decision-making software nowadays, it is important that the outcome of this process, called top-$k$ selection, is fair. Here we consider the problem of identifying a linear scoring function for top-$k$ selection that is fair. The function computes a score for each item as a weighted sum of its (numerical) attribute values. Additionally, the function must ensure that the subset selected is a faithful representative of the entire dataset for a minority or historically disadvantaged group. Existing algorithms do not scale effectively on large, high-dimensional datasets. Our theoretical analysis shows that in more than two dimensions, no algorithm is likely to achieve good scalability with respect to dataset size (i.e., a run time of $O(ncdot ext{polylog}(n))$), and the computational complexity is likely to increase rapidly with dimensionality. However, there are exceptions for small values of $k$ and for this case we provide significantly faster algorithms. We also provide efficient practical variants of these algorithms. Our implementations of these take advantage of modern hardware (e.g., exploiting parallelism). For large values of $k$, we give an alternative algorithm that, while theoretically worse, performs better in practice. Experimental results on real-world datasets demonstrate the efficiency of our proposed algorithms, which achieve speed-ups of up to several orders of magnitude compared to the state of the art (SoTA).