Counterfactual Strategies for Markov Decision Processes

2025-05-14

AI Summary

This paper addresses the insufficient safety of policies in Markov decision processes (MDPs), where an initial policy may yield undesirable outcomes with probability exceeding a prescribed safety threshold. To mitigate this, we propose Counterfactual Policy Optimization (CPO): given an unsafe policy, CPO computes the minimal perturbation to it such that the resulting policy satisfies probabilistic safety constraints. Unlike prior work limited to single-step counterfactual reasoning, ours is the first to extend counterfactual inference to multi-step policy spaces, enabling generation of diverse, interpretable, and safety-compliant policies. Methodologically, we formulate a nonlinear optimization problem jointly encoding policy perturbations, formal probabilistic verification, and diversity-promoting regularization. Experiments on four real-world datasets demonstrate that CPO significantly reduces the probability of adverse outcomes while ensuring safety, interpretability, and computational tractability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Counterfactuals are widely used in AI to explain how minimal changes to a model's input can lead to a different output. However, established methods for computing counterfactuals typically focus on one-step decision-making, and are not directly applicable to sequential decision-making tasks. This paper fills this gap by introducing counterfactual strategies for Markov Decision Processes (MDPs). During MDP execution, a strategy decides which of the enabled actions (with known probabilistic effects) to execute next. Given an initial strategy that reaches an undesired outcome with a probability above some limit, we identify minimal changes to the initial strategy to reduce that probability below the limit. We encode such counterfactual strategies as solutions to non-linear optimization problems, and further extend our encoding to synthesize diverse counterfactual strategies. We evaluate our approach on four real-world datasets and demonstrate its practical viability in sophisticated sequential decision-making tasks.

Problem

• Extend counterfactuals to sequential decision-making in MDPs
• Find minimal strategy changes to avoid undesired outcomes
• Encode and synthesize diverse counterfactual strategies via optimization

Innovation

• Introduces counterfactual strategies for MDPs
• Encodes strategies as non-linear optimization problems
• Synthesizes diverse counterfactual strategies