A Unified Benchmark for Privacy-preserving Vector Search

2026-08-02

AI Summary

This study addresses the lack of a standardized evaluation framework for privacy-preserving vector search schemes, which has hindered fair comparisons across performance, privacy guarantees, and recall—impeding real-world deployment. To bridge this gap, we establish the first unified benchmark enabling cross-scheme comparability by evaluating a plaintext baseline and four prominent encrypted approaches (SAP, EMVP, BNTM, Tiptoe) under identical workloads, hardware, and metrics. Our experiments uncover distinct Pareto trade-offs between privacy and efficiency: SAP achieves near-plaintext performance; EMVP provides cryptographic indistinguishability at the cost of a 4× throughput reduction; BNTM supports malicious-server verifiability with a 22× latency overhead; and Tiptoe conceals cluster selection at a query cost 190× higher than plaintext. Notably, GPU acceleration benefits only the plaintext and SAP implementations.

Abstract

Vector search powers semantic search, recommendation systems, and retrieval-augmented generation (RAG). By design, the service answering a query sees both the query embedding and, usually, the corpus against which it is matched. This is a privacy breach for both the user issuing the query and the owner of the corpus. A family of cryptographic schemes (e.g., SAP, EMVP, BNTM, Tip-toe) addresses that leak. However, as each scheme is published and evaluated on its own corpus, threat model, parameter choices, hardware, and metric conventions, the numbers cannot be compared directly. Consequently, a practitioner asking which one to deploy today has no defensible way to choose. We close that gap with a uniform experimental comparison, including a Plaintext baseline and four cryptographic backends running over the same workload, hardware, and metric definitions. Under that ruler, the schemes spread across a Pareto frontier in privacy, performance, and recall rather than imposing a flat penalty on performance. We find that the performance of SAP matches Plaintext, EMVP delivers cryptographic indistinguishability at a 4x throughput cost on CPU, BNTM adds malicious-server verifiability at a further 22x median-latency cost, and Tiptoe hides the cluster choice itself, but incurs a 190x per-query cost compared to Plaintext. GPU acceleration pays off for Plaintext and SAP but not for EMVP or BNTM. All our experiment artifacts are publicly available for reproducibility

Problem

• privacy-preserving vector search
• cryptographic schemes
• benchmarking
• retrieval-augmented generation
• semantic search

Innovation

• privacy-preserving vector search
• unified benchmark
• cryptographic protocols
• Pareto frontier
• reproducible evaluation