🤖 AI Summary
Event sequence (EvS) classification lacks standardized benchmarks and evaluation protocols, leading to incomparable model performance and unreliable conclusions. To address this, we introduce the first comprehensive benchmark specifically designed for EvS classification, comprising ten real-world and synthetic datasets—including the largest publicly available banking dataset to date—spanning domains such as healthcare and finance. We propose a standardized evaluation pipeline and release an open-source, extensible PyTorch library. Leveraging this framework, we systematically evaluate nine state-of-the-art sequential models (e.g., GRU, Transformer, Neural ODE), revealing for the first time fundamental modeling challenges unique to EvS—distinct from conventional time-series data. Empirical results demonstrate that GRU-based models achieve overall superior performance. This work establishes foundational infrastructure to advance reproducibility, fair benchmarking, and practical deployment of EvS classification methods.
📝 Abstract
Event Sequences (EvS) refer to sequential data characterized by irregular sampling intervals and a mix of categorical and numerical features. Accurate classification of these sequences is crucial for various real-life applications, including healthcare, finance, and user interaction. Despite the popularity of the EvS classification task, there is currently no standardized benchmark or rigorous evaluation protocol. This lack of standardization makes it difficult to compare results across studies, which can result in unreliable conclusions and hinder progress in the field. To address this gap, we present EBES, a comprehensive benchmark for EvS classification with sequence-level targets. EBES features standardized evaluation scenarios and protocols, along with an open-source PyTorch library that implements 9 modern models. Additionally, it includes the largest collection of EvS datasets, featuring 10 curated datasets, including a novel synthetic dataset and real-world data with the largest publicly available banking dataset. The library offers user-friendly interfaces for integrating new methods and datasets. Our benchmarking results highlight the unique properties of EvS compared to other sequential data types, provide a performance ranking of modern models with GRU-based models achieving the best results and reveal the challenges associated with robust EvS learning. The goal of EBES is to facilitate reproducible research, expedite progress in the field, and increase the real-world impact of EvS classification techniques.