What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control

2026-04-29

AI Summary

Large language models (LLMs) frequently deviate from Nash equilibrium in strategic games, yet the underlying mechanisms and their reversibility remain unclear. This study investigates Llama-3-8B and other models through self-play and cross-play experiments, employing neural probing, residual stream interventions, and a novel technique termed concept clamping. The work reveals, for the first time, that LLMs lack a dedicated internal module for Nash strategies; instead, such strategies are actively suppressed by later-stage prosocial mechanisms. Concept clamping enables causal control over game-theoretic behavior, demonstrating that smaller models exhibit impaired Nash compliance due to flawed chain-of-thought reasoning, whereas models of 70B parameters or larger nearly perfectly implement equilibrium strategies. Cross-play analyses further uncover mechanisms driving cooperation breakdown and equilibrium selection in coordination games.

Abstract

LLM agents are known to deviate from Nash equilibria in strategic interactions, but nobody has looked inside the model to understand why, or asked whether the deviation can be reversed. We do both. Working with four open-source models (Llama-3 and Qwen2.5, 8B to 72B parameters) playing four canonical two-player games, we establish the behavioral picture through self-play and cross-play experiments, then open up the 32-layer Llama-3-8B model and examine what actually happens during a strategic decision. The mechanistic findings are clear. Opponent history is encoded with near-perfect fidelity at the first layer (96% probe accuracy) and consumed progressively by later ones, while Nash action encoding is weak throughout, never exceeding 56%. There is no dedicated Nash module. Instead, the model privately favors the Nash action through most of its forward pass, but a prosocial override concentrated in the final layers reverses this, reaching 84% probability of cooperation at layer 30. When we inject a learned Nash direction into the residual stream, the behavior shifts bidirectionally, confirmed through concept clamping. The behavioral experiments surface six scale- and architecture-dependent findings, the most notable being that chain-of-thought reasoning worsens Nash play in small models but achieves near-perfect Nash play above 70B parameters. The cross-play experiments reveal three phenomena invisible in self-play: a small model can unravel any partner's cooperation by defecting early; two large models reinforce each other's cooperative instincts indefinitely; and who moves first in a coordination game determines which Nash equilibrium the system reaches. LLMs do not lack Nash-playing competence. They compute it, then suppress it.

Problem

• Nash equilibrium
• large language models
• strategic interaction
• behavioral deviation
• mechanistic interpretation

Innovation

• mechanistic interpretability
• Nash equilibrium suppression
• prosocial override
• concept clamping
• causal intervention