StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets

2025-07-12

AI Summary

Large language models (LLMs) lack high-fidelity evaluation frameworks for real-world financial decision-making. Method: We propose the first open-source, dual-mode, order-level financial simulation platform, systematically incorporating critical real-world constraints—including market latency, transaction slippage, and limit-order-book microstructure—and supporting both fine-grained (order-by-order) and coarse-grained (time-aggregated) simulation modes. Our approach employs a role-based, scalable multi-agent framework that couples a limit-order-driven market engine with realistic market dynamics modeling. Experiments enable high-fidelity assessment of LLM-driven trading strategies, particularly their sequential decision-making and uncertainty-aware reasoning capabilities. Contribution/Results: We introduce the first open-source benchmark platform tailored to finance, enabling heterogeneous strategy modeling and multi-agent collaboration. It significantly improves evaluation validity and reproducibility of LLMs in dynamic, incomplete-information environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We present StockSim, an open-source simulation platform for systematic evaluation of large language models (LLMs) in realistic financial decision-making scenarios. Unlike previous toolkits that offer limited scope, StockSim delivers a comprehensive system that fully models market dynamics and supports diverse simulation modes of varying granularity. It incorporates critical real-world factors, such as latency, slippage, and order-book microstructure, that were previously neglected, enabling more faithful and insightful assessment of LLM-based trading agents. An extensible, role-based agent framework supports heterogeneous trading strategies and multi-agent coordination, making StockSim a uniquely capable testbed for NLP research on reasoning under uncertainty and sequential decision-making. We open-source all our code at https: //github.com/harrypapa2002/StockSim.

Problem

• Evaluates LLMs in realistic financial decision-making scenarios
• Models market dynamics with real-world factors like latency
• Supports multi-agent coordination for trading strategies

Innovation

• Dual-mode order-level simulator for financial markets
• Incorporates real-world factors like latency and slippage
• Extensible role-based agent framework for diverse strategies