Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback

2025-11-12

AI Summary

In interactive preference elicitation (IPE), sparse human feedback severely limits the efficiency of dueling bandits (DB) algorithms. To address this, we propose a model-free DB framework that does not rely on a parametric reward model. Our approach features: (1) an enhanced human feedback mechanism extending beyond standard pairwise comparisons; (2) adaptive confidence bounds coupled with generalized concentration analysis to robustly model non-stationary, non-parametric preferences; and (3) a theoretically grounded regret bound accounting for multi-faceted trade-offs. Empirical evaluation across recommendation systems, multi-objective optimization, and response ranking for large language models demonstrates substantial improvements over state-of-the-art DB methods, validating both effectiveness and generalizability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Interactive preference elicitation (IPE) aims to substantially reduce human effort while acquiring human preferences in wide personalization systems. Dueling bandit (DB) algorithms enable optimal decision-making in IPE building on pairwise comparisons. However, they remain inefficient when human feedback is sparse. Existing methods address sparsity by heavily relying on parametric reward models, whose rigid assumptions are vulnerable to misspecification. In contrast, we explore an alternative perspective based on feedback augmentation, and introduce critical improvements to the model-free DB framework. Specifically, we introduce augmented confidence bounds to integrate augmented human feedback under generalized concentration properties, and analyze the multi-factored performance trade-off via regret analysis. Our prototype algorithm achieves competitive performance across several IPE benchmarks, including recommendation, multi-objective optimization, and response optimization for large language models, demonstrating the potential of our approach for provably efficient IPE in broader applications.

Problem

• Improving dueling bandits with augmented feedback to handle sparse human preferences
• Addressing inefficiency in preference elicitation without rigid parametric models
• Enhancing interactive decision-making for recommendation and language model optimization

Innovation

• Augmented confidence bounds integrate augmented human feedback
• Model-free dueling bandits framework with generalized concentration properties
• Multi-factored performance trade-off analyzed via regret analysis