The Matryoshka Hypencoder

2026-07-19

AI Summary

This work proposes a variable-size query neural network (Q-Nets) that integrates Hypencoder with Matryoshka representation learning to reduce the computational overhead of query encoding and improve deployment efficiency in retrieval systems. The approach introduces, for the first time, multi-granularity parameterization of Hypencoder, enabling efficient relevance scoring over precomputed document embeddings while offering flexible trade-offs between effectiveness and efficiency. Experimental results demonstrate that, under comparable in-domain retrieval performance, the method reduces active parameters by approximately 7× in-domain and by 50% out-of-domain, while achieving a 1.6–3.4× improvement in scoring throughput.

Abstract

The Hypencoder is a recently-proposed retrieval approach that encodes queries as shallow neural networks ("Q-Nets") that estimate relevance over pre-computed document embeddings. Inspired by Matryoshka Representation Learning, we show that the Hypencoder can be extended to support multiple sizes of Q-Nets, allowing trade-offs between effectiveness and efficiency when deployed. We find that this "Matryoshka Hypencoder" achieves comparable in-domain effectiveness with approximately 7x fewer active parameters in-domain and half as many active parameters out-of-domain, which corresponds to a 1.6-3.4x increase in scoring throughput. This work paves the way for practical deployment of Hypencoders.

Problem

• Hypencoder
• retrieval
• efficiency-effectiveness trade-off
• query encoding
• Matryoshka Representation

Innovation

• Matryoshka Representation Learning
• Hypencoder
• Query Encoding
• Efficiency-Effectiveness Trade-off
• Neural Retrieval