Progressive Content Refinement with Decaying Reward Joint LinUCB

2026-08-06

AI Summary

This work addresses the diminishing returns often observed in iterative optimization of large language models, which stem from over-exploitation due to neglecting reward saturation effects. To mitigate this issue, the authors propose a novel contextual bandit algorithm that integrates a reward decay mechanism into a unified LinUCB framework for the first time. The method employs an expectation-maximization (EM) algorithm to jointly learn prompt embeddings, arm values, and decay parameters, thereby dynamically adapting the prompt selection strategy to alleviate over-exploitation. Evaluated on the Sentiment Reversal and GSM8K benchmarks, the approach significantly outperforms strong baselines, and ablation studies confirm that explicitly modeling reward decay is crucial for performance gains.

Abstract

Iterative refinement has significantly enhanced Large Language Model (LLM) performance; however, existing methods ranging from feedback-based Self-Refine to traditional bandit approaches often rely on static options or overlook the saturation effect. This neglect leads to over-exploitation, where the continuous use of identical prompts or arms results in diminishing rewards over time. To address this challenge, we propose a novel contextual bandit algorithm that explicitly incorporates reward decay modeling. Utilizing an Expectation-Maximization (EM) algorithm, our method simultaneously estimates both arm-specific and decay parameters. Furthermore, by embedding prompts as arms, we facilitate the joint learning of arm values, distinguishing our approach from the traditional disjoint Linear Upper Confidence Bound (LinUCB) framework. Experimental results on Sentiment Reversal and GSM8K benchmarks demonstrate that our method achieves significant performance gains over strong baselines. Finally, our ablation study confirms that the integration of reward decay modeling within the bandit framework is crucial for mitigating over-exploitation and optimizing the iterative refinement process.

Problem

• iterative refinement
• reward decay
• over-exploitation
• contextual bandit
• Large Language Model

Innovation

• reward decay
• contextual bandits
• iterative refinement
• LinUCB
• EM algorithm