Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics

2025-09-15

AI Summary

This paper addresses the poor adaptability of conventional market-making strategies in non-stationary limit-order-book environments. Methodologically, we design a high-fidelity simulation environment incorporating empirically observed market phenomena—including clustered order arrivals, time-varying bid-ask spreads, return drifts, stochastic order sizes, and price volatility—and model order flow and price dynamics using stochastic control theory. We train a robust market-making agent via Proximal Policy Optimization (PPO). Our key contribution lies in explicitly encoding multiple market anomalies into the learning framework to enhance policy generalization under non-stationarity. Experimental results demonstrate that the proposed strategy significantly outperforms classical analytical solutions in profit stability, risk mitigation, and cross-market adaptability. The simulation platform proves effective for both training and pretraining RL-based market-making agents, validating its practical deployment potential.

Technology Category

• Multiagent Systems: Adversarial Agents
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Online Learning & Bandits

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent's performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Problem

• Develops reinforcement learning for market making optimization
• Addresses non-stationary limit order book dynamics challenges
• Creates adaptive trading agent for volatile market conditions

Innovation

• Reinforcement learning agent for market making
• Proximal-Policy Optimization algorithm implementation
• Simulator-based environment for non-stationary conditions