Multi-Agent Reinforcement Learning for Market Making: Competition without Collusion

2025-10-29

AI Summary

This study investigates the emergence mechanisms of algorithmic collusion—such as cartel formation or market dominance—and its impact on market efficiency in multi-agent AI trading. We propose a hierarchical reinforcement learning framework that integrates self-interested, competitive, and hybrid agents with tunable behavioral profiles, modeling dynamic market-making games under adversarial order flow. Innovatively, we design interaction-level metrics to quantify behavioral asymmetry and system stability, and develop a joint analytical model capturing bid-ask spread dynamics and order flow characteristics. Experimental results demonstrate that competitive agents significantly narrow bid-ask spreads and improve execution efficiency; hybrid agents achieve dominant market share while mitigating payoff erosion for other agents. These findings validate the efficacy of adaptive incentive regulation in suppressing collusive behavior and fostering sustainable coexistence among autonomous trading agents.

Technology Category

• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Mechanism Design
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Algorithmic collusion has emerged as a central question in AI: Will the interaction between different AI agents deployed in markets lead to collusion? More generally, understanding how emergent behavior, be it a cartel or market dominance from more advanced bots, affects the market overall is an important research question. We propose a hierarchical multi-agent reinforcement learning framework to study algorithmic collusion in market making. The framework includes a self-interested market maker (Agent~A), which is trained in an uncertain environment shaped by an adversary, and three bottom-layer competitors: the self-interested Agent~B1 (whose objective is to maximize its own PnL), the competitive Agent~B2 (whose objective is to minimize the PnL of its opponent), and the hybrid Agent~B$^star$, which can modulate between the behavior of the other two. To analyze how these agents shape the behavior of each other and affect market outcomes, we propose interaction-level metrics that quantify behavioral asymmetry and system-level dynamics, while providing signals potentially indicative of emergent interaction patterns. Experimental results show that Agent~B2 secures dominant performance in a zero-sum setting against B1, aggressively capturing order flow while tightening average spreads, thus improving market execution efficiency. In contrast, Agent~B$^star$ exhibits a self-interested inclination when co-existing with other profit-seeking agents, securing dominant market share through adaptive quoting, yet exerting a milder adverse impact on the rewards of Agents~A and B1 compared to B2. These findings suggest that adaptive incentive control supports more sustainable strategic co-existence in heterogeneous agent environments and offers a structured lens for evaluating behavioral design in algorithmic trading systems.

Problem

• Studying algorithmic collusion in market making using multi-agent reinforcement learning
• Analyzing how emergent behavior affects market outcomes and interaction patterns
• Evaluating adaptive incentive control for sustainable strategic co-existence

Innovation

• Hierarchical multi-agent reinforcement learning framework for markets
• Adversarial training with self-interested and competitive agents
• Interaction metrics quantify behavioral asymmetry and market dynamics