Inducing Diversity in Differentiable Search Indexing

2025-02-05

AI Summary

This work addresses the relevance-diversity trade-off in differentiable search indexing (DSI), proposing the first end-to-end DSI framework that explicitly incorporates diversity modeling into training. Unlike conventional approaches relying on post-hoc Maximal Marginal Relevance (MMR) reranking, we introduce an MMR-inspired diversity-regularized loss function jointly optimizing document relevance and information novelty during training. Built upon a Transformer architecture, our method natively supports diversity-sensitive tasks—such as subtopic retrieval—and enables efficient incremental index updates. Experiments on NQ320K and MSMARCO demonstrate substantial improvements in result diversity while preserving relevance performance; notably, high-quality, diverse recall lists are generated without any post-processing. Our approach thus bridges a critical gap in DSI by unifying relevance and diversity optimization within a single differentiable framework.

Technology Category

Search and Optimization: Distributed SearchData Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalMachine Learning: Learning Preferences or Rankings

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingGraph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Differentiable Search Indexing (DSI) is a recent paradigm for information retrieval which uses a transformer-based neural network architecture as the document index to simplify the retrieval process. A differentiable index has many advantages enabling modifications, updates or extensions to the index. In this work, we explore balancing relevance and novel information content (diversity) for training DSI systems inspired by Maximal Marginal Relevance (MMR), and show the benefits of our approach over the naive DSI training. We present quantitative and qualitative evaluations of relevance and diversity measures obtained using our method on NQ320K and MSMARCO datasets in comparison to naive DSI. With our approach, it is possible to achieve diversity without any significant impact to relevance. Since we induce diversity while training DSI, the trained model has learned to diversify while being relevant. This obviates the need for a post-processing step to induce diversity in the recall set as typically performed using MMR. Our approach will be useful for Information Retrieval problems where both relevance and diversity are important such as in sub-topic retrieval. Our work can also be easily be extended to the incremental DSI settings which would enable fast updates to the index while retrieving a diverse recall set.

Problem

• Balancing relevance and diversity in DSI
• Training DSI to diversify without post-processing
• Extending DSI for fast, diverse index updates

Innovation

• Transformer-based neural network indexing
• Diversity induction during DSI training
• No post-processing for diverse recall