DNQ: Deep Nash Q-Network for Partially Observable n-Player Games

2026-06-04

AI Summary

This work addresses the challenge of efficiently learning Nash equilibria in partially observable, simultaneous-move multi-agent games—such as repeated auctions and resource allocation settings—where traditional approaches suffer from intractable computational complexity. The authors propose the DNQ framework, which iterates through four phases: trajectory collection, shared critic–based payoff estimation, equilibrium computation via an external solver, and policy imitation. Its key innovation lies in replacing high-dimensional payoff tensors with a scalable pairwise payoff model, drastically reducing the complexity of equilibrium computation. By leveraging a shared critic, the method amortizes payoff learning across agents and states, maintaining strategic effectiveness while substantially lowering training overhead. This enables successful scaling to larger numbers of agents, a regime where conventional tensor-based methods rapidly become infeasible.

Abstract

Many real-world competitive systems require multiple decision-makers to act simultaneously under shared constraints, limited information, and repeated interaction, as in auctions, resource allocation, and security competition. We study multi-turn simultaneous bidding as a controlled testbed for such problems and propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents. DNQ alternates between trajectory collection, critic-based payoff estimation, equilibrium computation, and policy imitation. At each visited state, a shared critic predicts either pairwise payoff matrices or an exact N-player payoff tensor, an external solver computes equilibrium strategies, and the agents are trained by minimizing the KL divergence between their masked policies and the solver-derived equilibrium targets. We focus on a scalable pairwise formulation that greatly reduces equilibrium-solving cost and training time compared with the exact formulation, while the shared critic amortizes payoff learning across agents and states. Experiments compare the pairwise and exact variants using critic loss, policy entropy, bidding resource usage, and training cost, showing that the pairwise method scales to larger numbers of agents, whereas the exact method becomes computationally impractical as the joint game grows. These results illustrate the trade-off between strategic fidelity and scalability in repeated competitive environments.

Problem

• Partially Observable
• n-Player Games
• Simultaneous Bidding
• Equilibrium Computation
• Scalability

Innovation

• Deep Nash Q-Network
• solver-in-the-loop
• equilibrium supervision
• pairwise payoff formulation
• partially observable n-player games