FLAIR: Feedback Learning for Adaptive Information Retrieval

2025-08-18

AI Summary

To address insufficient adaptability of information retrieval in complex technical scenarios, this paper proposes a lightweight feedback learning framework that dynamically integrates domain-expert feedback to optimize retrieval strategies. Methodologically, it jointly models user feedback and document generation as an offline evaluation metric, and employs decentralized storage with a dual-channel ranking mechanism to online synergistically fuse semantic similarity and feedback signals—enabling adaptive retrieval optimization for both novel and historical queries. Technically, the framework integrates large language models, two-stage ranking, synthetic data generation, and a feedback-driven closed-loop iterative process. Evaluated in real-world industrial settings, it significantly outperforms baseline methods. Deployed in Microsoft’s Copilot DECO system, it serves thousands of users, demonstrating strong scalability, robustness, and practical effectiveness.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in Large Language Models (LLMs) have driven the adoption of copilots in complex technical scenarios, underscoring the growing need for specialized information retrieval solutions. In this paper, we introduce FLAIR, a lightweight, feedback learning framework that adapts copilot systems' retrieval strategies by integrating domain-specific expert feedback. FLAIR operates in two stages: an offline phase obtains indicators from (1) user feedback and (2) questions synthesized from documentation, storing these indicators in a decentralized manner. An online phase then employs a two-track ranking mechanism to combine raw similarity scores with the collected indicators. This iterative setup refines retrieval performance for any query. Extensive real-world evaluations of FLAIR demonstrate significant performance gains on both previously seen and unseen queries, surpassing state-of-the-art approaches. The system has been successfully integrated into Copilot DECO, serving thousands of users at Microsoft, demonstrating its scalability and effectiveness in operational environments.

Problem

• Adapting copilot retrieval strategies using expert feedback
• Improving information retrieval for specialized technical scenarios
• Combining similarity scores with feedback indicators iteratively

Innovation

• Lightweight feedback learning framework
• Two-track ranking mechanism integration
• Decentralized storage of expert indicators