D-NOVA: In-Storage Retrieval Accelerator via Dual-Bound 3D NAND-Optimized Similarity Search with Vector Adaptation

2026-07-20

AI Summary

Dense vector retrieval introduces significant latency and energy overhead in Retrieval-Augmented Generation (RAG), becoming a critical bottleneck in large language model inference. This work proposes a hardware-software co-design that deeply integrates inverted file (IVF)-based hierarchical retrieval into 3D NAND memory arrays, enabling fully in-storage vector similarity search. Key innovations include Dual-Bound Tight Similarity Sensing (DTS), a distance metric optimized for NAND strings, and a lightweight contrastive adapter that maps vectors into a DTS-friendly domain, achieving high recall while substantially improving energy efficiency. Compared to a CPU baseline, the system achieves up to 41.7× speedup and 71× higher energy efficiency; against state-of-the-art in-storage RAG accelerators, it delivers 12.13× higher throughput and 1.26× better energy efficiency.

Abstract

Retrieval-Augmented Generation (RAG) enhances the factual grounding of large language model (LLM) inference by retrieving relevant information from external knowledge bases. However, its dense vector retrieval introduces significant latency and energy overhead, becoming the primary performance bottleneck. Although recent in-storage accelerators aim to reduce data movement, they still rely on host or embedded processors outside the memory, where nearly 70% of the total retrieval time is spent. As a result, they cannot fully overcome the bandwidth limitations, leading to yet another memory bottleneck. To tackle these limitations, we present D-NOVA, a hardware-software co-designed in-storage retrieval accelerator. D-NOVA executes an inverted file (IVF)-based hierarchical retrieval pipeline by deeply embedding the search functionality directly into the NAND memory array. This is achieved by incorporating a new distance metric, Dual-Bound Tight Similarity Sensing (DTS), which is specifically tailored for searching within the NAND string. In addition, we introduce a lightweight contrastive adapter that maps embedding vectors into a DTS-friendly domain, recovering near-software recall while improving performance and energy efficiency. D-NOVA is up to 41.7x faster and 71x more energy-efficient than a CPU baseline, and achieves 12.13x higher throughput while being up to 1.26x more energy-efficient than state-of-the-art in-storage RAG accelerators, demonstrating the potential of fully in-storage vector search for scalable RAG acceleration.

Problem

• Retrieval-Augmented Generation
• vector retrieval
• in-storage acceleration
• memory bottleneck
• latency overhead

Innovation

• in-storage computing
• vector search
• 3D NAND
• similarity search
• Retrieval-Augmented Generation