A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits

2026-01-19

AI Summary

This work addresses the decision-making challenges faced by large language models (LLMs) in training, retrieval-augmented generation, and personalization, as well as the representational limitations of multi-armed bandits (MABs) in action definition and environment modeling. It presents the first systematic, component-level analysis of the bidirectional interaction mechanisms between LLMs and MABs. By establishing a component-wise analytical framework and conducting a comprehensive literature review, the study introduces a novel mutual enhancement paradigm—“LLM-enhanced MAB” and “MAB-enhanced LLM”—characterizing representative architectures, performance trends, and key challenges. To foster further research, the authors also release an open-access literature index repository.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have become powerful and widely used systems for language understanding and generation, while multi-armed bandit (MAB) algorithms provide a principled framework for adaptive decision-making under uncertainty. This survey explores the potential at the intersection of these two fields. As we know, it is the first survey to systematically review the bidirectional interaction between large language models and multi-armed bandits at the component level. We highlight the bidirectional benefits: MAB algorithms address critical LLM challenges, spanning from pre-training to retrieval-augmented generation (RAG) and personalization. Conversely, LLMs enhance MAB systems by redefining core components such as arm definition and environment modeling, thereby improving decision-making in sequential tasks. We analyze existing LLM-enhanced bandit systems and bandit-enhanced LLM systems, providing insights into their design, methodologies, and performance. Key challenges and representative findings are identified to help guide future research. An accompanying GitHub repository that indexes relevant literature is available at https://github.com/bucky1119/Awesome-LLM-Bandit-Interaction.

Problem

• Large Language Models
• Multi-Armed Bandits
• Component-Level Interaction
• Bidirectional Integration
• Adaptive Decision-Making

Innovation

• Large Language Models
• Multi-Armed Bandits
• Component-Level Interaction
• Retrieval-Augmented Generation
• Sequential Decision-Making