Deep Reinforcement Learning-Based Bidding Strategies for Prosumers Trading in Double Auction-Based Transactive Energy Market

2025-02-16

AI Summary

Modeling double-auction markets with stochastic prosumer bidding and distributed energy resource (DER) uncertainty remains challenging; existing reinforcement learning approaches suffer from scalability limitations, training instability, and inadequate privacy preservation. Method: This paper proposes a distributed deep reinforcement learning (DRL) bidding framework for multi-DER prosumers. It introduces a novel proximal policy optimization (PPO) agent with a dual-action space (price and quantity), integrated within a distributed training architecture that enables collaborative optimization while ensuring data locality. The framework jointly incorporates stochastic uncertainty modeling and double-auction market mechanics. Contribution/Results: Experiments demonstrate that the proposed method achieves superior trade-offs between social welfare and user comfort, outperforms state-of-the-art (SOTA) methods in bidding revenue, and exhibits strong robustness across diverse market conditions—without requiring centralized data collection or compromising participant privacy.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search
• Machine Learning: Distributed Machine Learning & Federated Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

With the large number of prosumers deploying distributed energy resources (DERs), integrating these prosumers into a transactive energy market (TEM) is a trend for the future smart grid. A community-based double auction market is considered a promising TEM that can encourage prosumers to participate and maximize social welfare. However, the traditional TEM is challenging to model explicitly due to the random bidding behavior of prosumers and uncertainties caused by the energy operation of DERs. Furthermore, although reinforcement learning algorithms provide a model-free solution to optimize prosumers' bidding strategies, their use in TEM is still challenging due to their scalability, stability, and privacy protection limitations. To address the above challenges, in this study, we design a double auction-based TEM with multiple DERs-equipped prosumers to transparently and efficiently manage energy transactions. We also propose a deep reinforcement learning (DRL) model with distributed learning and execution to ensure the scalability and privacy of the market environment. Additionally, the design of two bidding actions (i.e., bidding price and quantity) optimizes the bidding strategies for prosumers. Simulation results show that (1) the designed TEM and DRL model are robust; (2) the proposed DRL model effectively balances the energy payment and comfort satisfaction for prosumers and outperforms the state-of-the-art methods in optimizing the bidding strategies.

Problem

• Optimize prosumers' bidding in double auction energy markets
• Enhance scalability and privacy in reinforcement learning applications
• Balance energy payment and comfort for prosumers effectively

Innovation

• Deep Reinforcement Learning model
• Distributed learning execution
• Double auction market design