Generative vector search to improve pathology foundation models across multimodal vision-language tasks

2025-12-22

AI Summary

Traditional embedding-based retrieval methods inadequately model biomedical multi-concept queries—e.g., coupled molecular, cellular, and physiological features across omics data and clinical reports. Method: We propose a generative vector search framework centered on Stochastic Implicit Matching (STHLM), which expands retrieval coverage via latent-space sampling conditioned on text or image inputs, enabling a “search broader” paradigm. The method integrates multimodal embedding alignment with a retrieval-augmented generation (RAG) architecture. Contribution/Results: Evaluated on three benchmarks—scientific literature, clinical notes, and histopathological images—the framework improves retrieval accuracy by 10–30% while compressing embedding dimensionality to one-tenth of baseline. Crucially, it enables test-time trade-offs: lightweight computation yields measurable accuracy gains—a first for multimodal pathology retrieval and a significant breakthrough in overcoming its long-standing bottlenecks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval-augmented generation improves large language models by grounding outputs in external knowledge sources, reducing hallucinations and addressing knowledge cutoffs. However, standard embedding-based retrieval fails to capture the complexity of multi-concept queries, particularly in domains like biomedicine, where biological data are inherently high-dimensional. For example,omics datasets, and clinical reports simultaneously exhibit numerous molecular, cellular, and physiological features. We present Stochastic Latent Matching (STHLM), a generative vector search method that samples query-conditioned embeddings from text or image inputs to enhance retrieval performance. Analogous to how Chain-of-Thought reasoning enables language models to "think longer" on complex problems, STHLM allows retrieval systems to "search wider" through iterative sampling. STHLM demonstrates critical improvements over classical vector retrieval across diverse benchmarks, including scientific literature, clinical notes, and tissue images, boosting retrieval performance by 10-30% through test-time compute (trading latency for accuracy), while enabling up to a 10-fold compression of embedding dimensions.

Problem

• Improves retrieval for complex multi-concept biomedical queries
• Enhances pathology foundation models across multimodal tasks
• Addresses limitations of standard embedding-based retrieval in biomedicine

Innovation

• Generative vector search method for multimodal retrieval
• Sampling query-conditioned embeddings from text or images
• Improves retrieval performance via iterative sampling and compression