Human-AI Collaboration with Misaligned Preferences

2025-11-04

Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization

AI Summary

This paper investigates collaborative recommendation under preference misalignment between humans and AI: when algorithms act as filters and humans serve as final decision-makers, preference noise, individual heterogeneity, and algorithmic recognition lag induce systematic misalignment. Contrary to conventional approaches that seek strict preference alignment, we demonstrate that *moderate* preference misalignment can enhance human utility—by introducing algorithmically induced, diverse biases that serendipitously increase the visibility and selection probability of high-value options. We propose a unified framework grounded in a noisy ranking model and mixed-integer programming (MIP), jointly optimizing utilitarian welfare (utility maximization) and developmental welfare (individual capability growth). For the former, we design an efficient near-optimal algorithm for the NP-hard problem; for the latter, we construct, for the first time, a deterministic algorithm satisfying developmental welfare constraints under specific conditions. Our results uncover a novel “beneficial misalignment” mechanism in multi-algorithm collaboration.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In many real-life settings, algorithms play the role of assistants, while humans ultimately make the final decision. Often, algorithms specifically act as curators, narrowing down a wide range of options into a smaller subset that the human picks between: consider content recommendation or chatbot responses to questions with multiple valid answers. This type of role is one of the most common in human-algorithm systems, such as content recommendation, search, and some types of categorical prediction. For instance, when a user requests directions, Google Maps returns a small set of routes, and the user typically picks her final route from those routes presented. Crucially, humans may not know their own preferences perfectly either, but instead may only have access to a noisy sampling over preferences. Algorithms can assist humans by curating a smaller subset of items, but must also face the challenge of misalignment: humans may have different preferences from each other (and from the algorithm), and the algorithm may not know the exact preferences of the human they are facing at any point. In this paper, we model and theoretically study such a setting. There are two actors: an algorithm and a human in the human-algorithm collaboration model. A set of items M = {x1, …, xm} representing different outcomes (e.g., labels in prediction tasks, generative model output). The specific mode of interaction we assume is where the algorithm picks their top k from some noisy ranking over items, and the human picks their favorite among that set, according to their own noisy ranking over items. Our first finding is that there are settings where misalignment can be helpful. We consider the setting where the human may work with multiple algorithms with different ground-truth. We show instances where the human benefit by collaborating with a misaligned algorithm. Surprisingly, we show that humans gain more utility from a misaligned algorithm (which makes different mistakes) than from an aligned algorithm. The key insight is that appropriate misalignment on low-valued items can increase the likelihood that humans select high-valued items. Next, we build the algorithm by studying two welfare objectives: utilitarian welfare and uplift. Social welfare measures the expected utility of the human, while uplift requires each type of human to benefit from the collaboration. While these two objectives do not always align with each other (i.e., there exist instances where the utilitarian welfare maximizing algorithm does not achieve uplift and vice versa), we design different approaches for achieving these two objectives. For utilitarian welfare maximization, we show that it is NP-hard and design an MIP that solves it effectively in practice. Meanwhile, although an uplift algorithm may not always exist in general, we provide some special scenarios where a naturally designed algorithm can always achieve uplift. We conclude by discussing implications for designers of algorithmic tools and policymakers.

Problem

• Modeling human-AI collaboration with misaligned preferences
• Studying how humans benefit from algorithms making different mistakes
• Identifying algorithm properties that maximize human welfare

Innovation

• Modeling human-AI collaboration with misaligned preferences
• Showing benefits from misaligned algorithms over aligned ones
• Studying algorithm properties that maximize human welfare