Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

2026-07-25

AI Summary

This work investigates how to train models in an unsupervised setting to achieve no-regret and swap-regret properties from game theory, thereby inducing equilibrium behavior. The authors introduce a novel approach by directly formulating external regret and swap regret as differentiable loss functions and embedding them within a single-layer self-attention architecture. This design ensures that forward propagation is equivalent to smoothed fictitious play and its swap variant. The framework naturally recovers the update rules of classical online learning algorithms such as those of Blum and Mansour, accurately simulating regret dynamics without explicit supervision. Consequently, it guarantees convergence to coarse correlated equilibria and even correlated equilibria, revealing a profound connection between attention mechanisms and game-theoretic equilibrium concepts.

Abstract

We revisit the regret loss framework introduced in Park et al. (2025), which uses decision-theoretic regret as a direct loss function for training models to make better decisions, through the lens of probability-simplex policies. Our first result shows that a single-layer self-attention model trained with regret loss admits a stationary point whose forward-pass exactly matches smoothed fictitious play with the appropriate stepsize that ensures no-regret behavior-i.e., for any given policy input, the model outputs the same update that smoothed fictitious play would produce. In parallel, we also newly introduce a swap-regret loss function, which extends the regret-loss framework beyond external regret and enables models to directly optimize for swap-deviation robustness. We further show that this swap-regret loss admits a stationary point whose forward pass implements the corresponding swap-regret update induced by classical Blum-Mansour no-pass implementation algorithm, with each head implementing an external-regret update via smoothed fictitious play. Together, these results show that regret-trained attention can realize differentiable mechanisms whose deployment induces equilibrium behavior in games: external-regret dynamics lead to coarse correlated equilibrium, while swap-regret dynamics lead to correlated equilibrium. Thus, regret-based objectives steer minimal attention architectures toward online-learning dynamics with game-theoretic guarantees, without supervised traces of those algorithms.

Problem

• regret loss
• self-attention
• no-regret learning
• correlated equilibrium
• probability simplex

Innovation

• regret loss
• swap regret
• self-attention
• smoothed fictitious play
• correlated equilibrium