Prioritized Ranking Experimental Design Using Recommender Systems in Two-Sided Platforms

2025-02-13

AI Summary

In online two-sided markets, estimating the treatment effect on the treated (TATE) for item-side interventions is often biased due to interference among users. To address this, we propose Two-Sided Prioritized Ranking (TSPR), the first experimental design that leverages a recommendation system as a causal intervention vehicle: it dynamically adjusts item ranking priority in search results based on item-level treatment status, explicitly modeling and mitigating cross-user interference while preserving user access completeness and treatment consistency. Integrating causal inference, experimental design, and recommender modeling, TSPR is evaluated via simulation on real-world search impression data from an online travel platform. Results demonstrate that TSPR accurately recovers the true TATE and significantly reduces overestimation error compared to conventional randomization baselines. This work establishes a scalable, low-intrusion paradigm for causal experimentation on two-sided platforms.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Search and Optimization: Algorithm Configuration

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

Interdependencies between units in online two-sided marketplaces complicate estimating causal effects in experimental settings. We propose a novel experimental design to mitigate the interference bias in estimating the total average treatment effect (TATE) of item-side interventions in online two-sided marketplaces. Our Two-Sided Prioritized Ranking (TSPR) design uses the recommender system as an instrument for experimentation. TSPR strategically prioritizes items based on their treatment status in the listings displayed to users. We designed TSPR to provide users with a coherent platform experience by ensuring access to all items and a consistent realization of their treatment by all users. We evaluate our experimental design through simulations using a search impression dataset from an online travel agency. Our methodology closely estimates the true simulated TATE, while a baseline item-side estimator significantly overestimates TATE.

Problem

• Mitigates interference bias in two-sided marketplaces
• Estimates total average treatment effect accurately
• Uses recommender system for experimental design

Innovation

• Recommender system prioritizes items
• Mitigates interference bias effectively
• Ensures coherent user platform experience