Improving the Efficiency and Effectiveness of LLM Knowledge Distillation for Conversational Search

2026-06-03

AI Summary

This work addresses the challenge of balancing efficiency and effectiveness in conversational search, where large language models (LLMs) can significantly enhance retrieval performance through query rewriting but incur substantial inference overhead. The study presents the first systematic investigation into knowledge distillation–based lightweight query rewriting, introducing a novel distillation objective that combines Kullback–Leibler divergence with contrastive loss, along with sparsity regularization to improve representation efficiency. Evaluated on the TopiOCQA dataset, the proposed method reduces inference FLOPs by 2× while degrading Recall@100 by less than 2%, markedly enhancing the practicality of first-stage retrieval and establishing a new paradigm for efficient distillation in conversational search.

Abstract

Conversational Search (CS) considers retrieval of relevant documents based on conversational context. Large Language Models (LLMs) have significantly enhanced CS by enabling effective query rewriting. However, employing LLMs during inference poses efficiency challenges. A method to balance effectiveness and efficiency is the use of knowledge distillation from LLM-based query rewriting. Recent work applies the Kullback-Leibler Divergence (KLD) for distillation, relaxing the alignment with the teacher signal compared to previous methods. Despite these gains, several aspects of KLD-based distillation for conversational search remain understudied, and we investigate them in this work. Prior work in related fields suggests that adding a contrastive loss to the KLD objective can improve performance; we confirm this and observe significant gains in precision-oriented ranking metrics. We also find that contrastive sampling strategies for the KLD loss have a non-trivial impact and must be chosen carefully. Although theory suggests that more samples improve the KLD estimate, experiments show diminishing returns on the number of used samples. Finally, we address the phenomenon of decreased sparsity in longer conversations, which limits computational efficiency across sparse retrieval methods. We find that the representations from the model distilled with the KLD loss can be strongly regularized with a regularization loss, substantially improving sparsity and inference efficiency without significantly harming retrieval effectiveness. We achieve a $2\times$ decrease in FLOPS on TopiOCQA with negligible loss in effectiveness, corresponding to a $\leq 2%$ drop in Recall@100. Our results provide insights into distillation objectives for learned sparse conversational retrievers and offer practical guidelines for improving effectiveness and efficiency in first-stage retrieval.

Problem

• Conversational Search
• Knowledge Distillation
• Query Rewriting
• Sparse Retrieval
• Efficiency-Effectiveness Trade-off

Innovation

• knowledge distillation
• conversational search
• contrastive loss
• sparse retrieval
• regularization