🤖 AI Summary
This work addresses the lack of a standardized backtesting framework in prediction markets, which hinders fair evaluation of trading agents under realistic market microstructure, transaction fees, and settlement risk. We propose the first event-driven replay system tailored for prediction markets, integrating limit order book simulation, a maker/taker fee model, and an interface for tool-augmented large language model agents. Inspired by SWE-bench, we introduce a standardized evaluation paradigm for this domain. Using data from the Kalshi platform, we construct four backtesting scenarios spanning cryptocurrency, weather, and sports events. Empirical results demonstrate that fee-aware strategies significantly outperform naive agents in high-volatility environments, as the latter suffer substantial losses due to neglecting transaction costs and settlement risks.
📝 Abstract
Prediction markets offer a natural testbed for trading agents: contracts have binary payoffs, prices can be interpreted as probabilities, and realized performance depends critically on market microstructure, fees, and settlement risk. We introduce PredictionMarketBench, a SWE-bench-style benchmark for evaluating algorithmic and LLM-based trading agents on prediction markets via deterministic, event-driven replay of historical limit-order-book and trade data. PredictionMarketBench standardizes (i) episode construction from raw exchange streams (orderbooks, trades, lifecycle, settlement), (ii) an execution-realistic simulator with maker/taker semantics and fee modeling, and (iii) a tool-based agent interface that supports both classical strategies and tool-calling LLM agents with reproducible trajectories. We release four Kalshi-based episodes spanning cryptocurrency, weather, and sports. Baseline results show that naive trading agents can underperform due to transaction costs and settlement losses, while fee-aware algorithmic strategies remain competitive in volatile episodes.