Adaptive Semantic Prompt Caching with VectorQ

2025-02-06

AI Summary

Existing semantic prompt caching systems employ static similarity thresholds, which fail to accommodate the inherent complexity and uncertainty across diverse embedding spaces—leading to low hit rates and high misapplication rates. To address this, we propose the Embedding-Aware Adaptive Threshold Mechanism (EATM), the first approach to dynamically learn personalized threshold regions for each query embedding, thereby abandoning the restrictive assumption of a uniform global threshold. EATM models semantic similarity via vector embeddings and jointly estimates differentiable, learnable threshold boundaries. We establish a multi-dimensional evaluation framework across four heterogeneous datasets. Experiments demonstrate that EATM substantially improves caching efficiency and reliability: it achieves up to a 12× increase in cache hit rate and reduces error rates by up to 92%. Overall, EATM provides a more robust and generalizable decision-making paradigm for LLM inference caching.

Technology Category

• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Semantic prompt caches reduce the latency and cost of large language model (LLM) inference by reusing cached LLM-generated responses for semantically similar prompts. Vector similarity metrics assign a numerical score to quantify the similarity between an embedded prompt and its nearest neighbor in the cache. Existing systems rely on a static threshold to classify whether the similarity score is sufficiently high to result in a cache hit. We show that this one-size-fits-all threshold is insufficient across different prompts. We propose VectorQ, a framework to learn embedding-specific threshold regions that adapt to the complexity and uncertainty of an embedding. Through evaluations on a combination of four diverse datasets, we show that VectorQ consistently outperforms state-of-the-art systems across all static thresholds, achieving up to 12x increases in cache hit rate and error rate reductions up to 92%.

Problem

• Optimize semantic prompt caching efficiency
• Adapt thresholds for embedding similarity
• Enhance LLM inference performance

Innovation

• Adaptive threshold learning
• Vector similarity metrics
• Semantic prompt caching