PolySim: Deterministic Polynomial Surrogates for Cross-Modal Retrieval on CiM

2026-07-22

AI Summary

This work addresses the sampling gap that arises when deploying probabilistic embedding–based cross-modal retrieval on compute-in-memory (CiM) hardware, where Monte Carlo sampling and nonlinear distance computations are incompatible with CiM’s support for only deterministic, single-step matrix-vector multiplication. To bridge this gap, the authors propose PolySim, a framework that approximates Gaussian distribution embeddings using low-order polynomial bases and introduces a learnable order-bilinear kernel to compute similarity, thereby enabling a fully deterministic retrieval pipeline. PolySim is the first method to achieve efficient probabilistic cross-modal retrieval on conventional crossbar-based CiM architectures, attaining up to a 10.3% improvement in R@1 across six video, image, and audio benchmarks—matching or surpassing PCME—while demonstrating strong robustness under device non-idealities in CrossSim evaluations.

Abstract

Cross-modal retrieval on edge devices benefits from probabilistic embeddings that capture semantic uncertainty, but deploying them on compute-in-memory (CiM) hardware remains an open problem. The core difficulty is a sampling gap: probabilistic methods such as PCME rely on Monte Carlo sampling and nonlinear distance evaluation at inference, which are fundamentally incompatible with CiM crossbar arrays that support only deterministic, single-step matrix-vector multiplication. Few existing probabilistic retrieval methods can be executed on a conventional crossbar. To bridge this gap, we propose PolySim, a framework that reformulates probabilistic retrieval into a fully deterministic pipeline. PolySim approximates each Gaussian embedding dimension using low-order polynomial bases and computes similarity via a learnable order-bilinear kernel, eliminating stochastic sampling while preserving distributional information. In experiments on six benchmarks spanning video, image, and audio retrieval, PolySim improves R@1 over deterministic baselines by up to 10.3\% and matches or exceeds PCME, while reducing inference to a single crossbar-compatible matrix-vector multiplication. CrossSim evaluation under realistic device non-idealities confirms robust deployment on conventional crossbar arrays. To the best of our knowledge, PolySim is the first method to enable probabilistic cross-modal retrieval on CiM hardware.

Problem

• cross-modal retrieval
• compute-in-memory
• probabilistic embeddings
• CiM hardware
• sampling gap

Innovation

• PolySim
• cross-modal retrieval
• compute-in-memory
• probabilistic embeddings
• deterministic approximation