DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection

2025-08-25

AI Summary

Time-series anomaly detection faces critical challenges including label scarcity, high false-positive rates, and poor generalization. This paper proposes a novel semi-supervised framework integrating reinforcement learning, variational autoencoders (VAEs), and active learning. Its core contribution is a dynamic reward scaling mechanism that adaptively balances reconstruction error and classification feedback, significantly enhancing detection robustness and accuracy under low-labeling regimes. This mechanism mitigates reward sparsity and improves generalization to previously unseen anomaly types. Evaluated on the Yahoo A1/A2 benchmark datasets, the method achieves substantial improvements in both precision and recall over state-of-the-art unsupervised and semi-supervised approaches, demonstrating its effectiveness and practical applicability.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Anomaly detection in time series data is important for applications in finance, healthcare, sensor networks, and industrial monitoring. Traditional methods usually struggle with limited labeled data, high false-positive rates, and difficulty generalizing to novel anomaly types. To overcome these challenges, we propose a reinforcement learning-based framework that integrates dynamic reward shaping, Variational Autoencoder (VAE), and active learning, called DRTA. Our method uses an adaptive reward mechanism that balances exploration and exploitation by dynamically scaling the effect of VAE-based reconstruction error and classification rewards. This approach enables the agent to detect anomalies effectively in low-label systems while maintaining high precision and recall. Our experimental results on the Yahoo A1 and Yahoo A2 benchmark datasets demonstrate that the proposed method consistently outperforms state-of-the-art unsupervised and semi-supervised approaches. These findings show that our framework is a scalable and efficient solution for real-world anomaly detection tasks.

Problem

• Detecting anomalies in time series with limited labeled data
• Reducing high false-positive rates in anomaly detection systems
• Generalizing reinforcement learning to novel anomaly types

Innovation

• Dynamic reward scaling mechanism
• VAE-based reconstruction error integration
• Active learning with adaptive rewards