When More Cores Hurts: The Vector Database Scaling Paradox in HPC

2026-06-07

AI Summary

This study addresses the limited scalability of existing vector databases in high-performance computing (HPC) systems, which hinders their ability to support scientific AI workloads. Conducting large-scale evaluations of Qdrant, Milvus, and Weaviate on two production supercomputers, the authors examine representative scientific access patterns—including mixed read-write and write-then-read scenarios—and uncover a counterintuitive scaling paradox: performance degrades as core count increases. Through multi-node distributed deployments using real-world scientific datasets and multimodal embeddings, coupled with parallel performance profiling, they demonstrate that query throughput drops by up to 30.67% with additional cores, and a 16× increase in node count yields only a 5.46× performance gain. These findings reveal a fundamental mismatch between current vector database architectures and HPC system design.

Abstract

Vector databases have been designed and optimized for cloud environments; however, emerging scientific AI workloads (e.g., molecular search, meteorological trajectory detection, and literature-driven hypothesis generation) demand efficient, scalable execution on HPC systems. We present a large-scale evaluation of three state-of-the-art vector databases -- Qdrant, Milvus, and Weaviate -- on two production supercomputers, scaling to 256 distributed workers across 64 compute nodes. We evaluate representative workload patterns -- mixed read/write and write-then-read -- using popular benchmarks, multimodal embeddings, and a novel real-world scientific dataset. Our results reveal that workload characteristics can limit latency reduction, additional cores can reduce query throughput by up to 30.67%, and scaling from 16 to 256 workers (16x) only yields a 5.46x improvement. This scaling paradox exposes the fundamental mismatch between cloud-oriented designs and HPC systems, highlighting the need for new, HPC-aware vector database designs.

Problem

• vector database
• HPC
• scaling paradox
• scientific AI workloads
• query throughput

Innovation

• vector database
• HPC
• scaling paradox
• performance evaluation
• distributed computing