Towards Human-AI Complementarity in Matching Tasks

2025-08-18

AI Summary

Existing data-driven matching systems fail to achieve human-AI complementarity—human-AI collaborative decisions do not significantly outperform either purely human or purely algorithmic decisions. Method: We propose Comatch, a confidence-aware matching system that dynamically allocates decision responsibility: high-confidence matches are executed automatically by AI, while low-confidence matches are delegated to human judgment. Contribution/Results: Comatch is the first system to theoretically and empirically validate the optimality of human-AI responsibility allocation, establishing a provably performance-improving collaboration mechanism. It integrates calibrated confidence estimation, adaptive delegation policies, and large-scale online randomized controlled trials (N=800). Results demonstrate statistically significant improvements in both matching accuracy and fairness over standalone human or algorithmic baselines, providing rigorous empirical support for the feasibility and efficacy of human-AI complementary advantage.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Learning Preferences or Rankings

Application Category

• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Data-driven algorithmic matching systems promise to help human decision makers make better matching decisions in a wide variety of high-stakes application domains, such as healthcare and social service provision. However, existing systems are not designed to achieve human-AI complementarity: decisions made by a human using an algorithmic matching system are not necessarily better than those made by the human or by the algorithm alone. Our work aims to address this gap. To this end, we propose collaborative matching (comatch), a data-driven algorithmic matching system that takes a collaborative approach: rather than making all the matching decisions for a matching task like existing systems, it selects only the decisions that it is the most confident in, deferring the rest to the human decision maker. In the process, comatch optimizes how many decisions it makes and how many it defers to the human decision maker to provably maximize performance. We conduct a large-scale human subject study with $800$ participants to validate the proposed approach. The results demonstrate that the matching outcomes produced by comatch outperform those generated by either human participants or by algorithmic matching on their own. The data gathered in our human subject study and an implementation of our system are available as open source at https://github.com/Networks-Learning/human-AI-complementarity-matching.

Problem

• Addressing lack of human-AI complementarity in matching decisions
• Optimizing confidence-based decision allocation between humans and algorithms
• Improving matching outcomes beyond human-only or algorithm-only approaches

Innovation

• Collaborative matching system defers decisions
• Optimizes decision allocation for performance
• Validated through large-scale human study