On the Robustness of Generative Information Retrieval Models

2024-12-25

AI Summary

This work systematically evaluates the out-of-distribution (OOD) robustness of generative information retrieval (IR) models under four realistic challenges: query perturbations, unseen query types, cross-task transfer, and corpus expansion. Method: We propose the first taxonomy for OOD robustness in generative IR and design a multi-dimensional evaluation protocol using state-of-the-art generative models (e.g., GENRE, RAG) and dense baselines (e.g., ColBERT, ANCE). Contribution/Results: Empirical results demonstrate that current generative IR models suffer substantial performance degradation across all OOD scenarios and consistently underperform dense retrievers in robustness. The study identifies critical generalization limitations of generative IR, establishes a theoretical framework for analyzing its reliability, and provides empirical foundations for developing trustworthy, robust retrieval systems.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Generative information retrieval methods retrieve documents by directly generating their identifiers. Much effort has been devoted to developing effective generative IR models. Less attention has been paid to the robustness of these models. It is critical to assess the out-of-distribution (OOD) generalization of generative IR models, i.e., how would such models generalize to new distributions? To answer this question, we focus on OOD scenarios from four perspectives in retrieval problems: (i)query variations; (ii)unseen query types; (iii)unseen tasks; and (iv)corpus expansion. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of representative generative IR models against dense retrieval models. Our empirical results indicate that the OOD robustness of generative IR models is in need of improvement. By inspecting the OOD robustness of generative IR models we aim to contribute to the development of more reliable IR models. The code is available at url{https://github.com/Davion-Liu/GR_OOD}.

Problem

• Generative Information Retrieval Models
• Query Variability
• Data Scalability

Innovation

• Generative Information Retrieval
• Robustness Evaluation
• Adaptability Analysis