Let them have CAKES: A Cutting-Edge Algorithm for Scalable, Efficient, and Exact Search on Big Data

2023-09-11

AI Summary

Exact k-NN search in large-scale, high-dimensional data suffers from poor efficiency, limited scalability, and difficulty adapting to arbitrary—especially non-metric—distance functions. Method: We propose three generic, exact k-NN algorithms grounded in metric geometry and the manifold hypothesis, introducing novel pruning and candidate filtering mechanisms. Their time complexity depends on the metric entropy and fractal dimension of the data—not on its cardinality or ambient dimension—enabling principled support for non-Euclidean distances such as Levenshtein and DTW. Contribution/Results: Implemented in Rust for high performance, our approach achieves >10× faster indexing over state-of-the-art methods on ANN-Benchmarks, genomic, and RF datasets; guarantees 100% recall in metric spaces; significantly outperforms SOTA in non-metric spaces; and exhibits near-constant scaling with data size.

Abstract

The ongoing Big Data explosion has created a demand for efficient and scalable algorithms for similarity search. Most recent work has focused on extit{approximate} $k$-NN search, and while this may be sufficient for some applications, extit{exact} $k$-NN search would be ideal for many applications. We present CAKES, a set of three novel, exact algorithms for $k$-NN search. CAKES's algorithms are generic over extit{any} distance function, and they extit{do not} scale with the cardinality or embedding dimension of the dataset, but rather with its metric entropy and fractal dimension. We test these claims on datasets from the ANN-Benchmarks suite under commonly-used distance functions, as well as on a genomic dataset with Levenshtein distance and a radio-frequency dataset with Dynamic Time Warping distance. We demonstrate that CAKES exhibits near-constant scaling with cardinality on data conforming to the manifold hypothesis, and has perfect recall on data in extit{metric} spaces. We also demonstrate that CAKES exhibits significantly higher recall than state-of-the-art $k$-NN search algorithms when the distance function is not a metric. Additionally, we show that indexing and tuning time for CAKES is an order of magnitude, or more, faster than state-of-the-art approaches. We conclude that CAKES is a highly efficient and scalable algorithm for exact $k$-NN search on Big Data. We provide a Rust implementation of CAKES under an MIT license at https://github.com/URI-ABD/clam

Problem

• Big Data
• K-Nearest Neighbor Search
• Efficient Algorithm

Innovation

• CAKES
• k-Nearest Neighbor Search
• Data Complexity Adaptive