Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

2026-07-24

AI Summary

This work addresses the challenges of high evaluation bias and unstable learning in existing off-policy evaluation and learning methods under realistic settings such as small-sample regimes, deterministic logging policies, and novel actions. To overcome these limitations, we propose the first cross-domain off-policy evaluation and learning framework tailored for contextual bandits. By jointly modeling historical logged data from the target domain and multiple related source domains, we introduce a novel cross-domain estimator and a corresponding policy gradient algorithm that effectively mitigate insufficient exploration and high-variance issues. Experimental results demonstrate that our approach significantly improves both policy evaluation accuracy and learning performance, consistently outperforming state-of-the-art single-domain methods across small-sample, deterministic logging, and new-action scenarios.

Abstract

Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the target domain in which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations.

Problem

• Off-Policy Evaluation
• Contextual Bandits
• Cross-Domain Learning
• Few-Shot Data
• Deterministic Logging Policies

Innovation

• Cross-Domain OPE/L
• Contextual Bandits
• Off-Policy Evaluation
• Policy Gradient
• Transfer Learning