Regret Minimization with Adaptive Opponents in Repeated Games

2026-06-04

AI Summary

This work addresses the limitation of traditional external regret in repeated games, which fails to capture scenarios where opponents adaptively adjust their strategies based on history. The authors propose a novel regret measure, RP-Regret, that quantifies the gap between a player’s actual cumulative utility against an adaptive opponent and the utility achievable by the best ex post response. This measure enables a stronger performance benchmark under weaker assumptions about opponent behavior and can lead to more efficient equilibria. To handle the non-convexity inherent in RP-Regret minimization, the paper integrates techniques from non-convex optimization, linearized surrogate objectives, optimization oracles, and direct optimization under the assumption of slowly changing opponents. Theoretically, RP-Regret is shown to grow sublinearly under suitable conditions, and experiments demonstrate that the approach effectively learns high-payoff cooperative equilibria in games such as Stag-Hunt.

Abstract

In this paper, we study regret minimization in repeated games with \emph{adaptive} opponents who can respond based on histories of play. The standard metric of \emph{external regret} in online learning is known to fail to capture such adaptivity. To account for players' counterfactual reasoning, we introduce {\tt Repeated Policy Regret (RP-Regret)}, a game-theoretic metric that measures the difference between the \emph{realized} and the \emph{best-in-hindsight} accumulated utility when all players can \emph{respond} to the history of play. Compared to existing regret notions in this setting, ours is native to repeated game playing, enabling stronger comparators and opponents with fewer constraints, while maintaining the possibility of finding better equilibria when all players minimize it. We first identify necessary conditions for obtaining {\tt RP-Regret} sublinear in time, on the variation of the player's comparator strategies in the regret definition and on the memories of both the comparator and opponents' strategies. We then study additional conditions and provable algorithms to minimize {\tt RP-Regret}, which is by definition \emph{non-convex} in the strategy space. To address this challenge, we propose three algorithms: (i) one based on an optimization oracle, as assumed in some prior work in online non-convex learning; (ii) one that minimizes a convex and \emph{linearized} surrogate of {\tt RP-Regret} at each iteration; (iii) one that directly minimizes {\tt RP-Regret} when opponents change strategies slowly. Furthermore, when all players can run algorithms to minimize the {\tt RP-Regret} (or its linearized variant), certain subgame perfect equilibria of the repeated game can be learned. We also provide experiments showing that minimizing our regret notions can lead to more cooperative solutions with higher utility in games such as Stag-Hunt.

Problem

• regret minimization
• repeated games
• adaptive opponents
• policy regret
• non-convex optimization

Innovation

• Repeated Policy Regret
• adaptive opponents
• non-convex online learning
• subgame perfect equilibrium
• regret minimization