Rethinking BPS: A Utility-Based Evaluation Framework

2025-05-28

AI Summary

Current BPS model evaluation suffers from two critical flaws: (1) conflating simulation modeling with predictive tasks, thereby failing to assess how well models capture actual process behavior; and (2) overreliance on global distance metrics—e.g., Earth Mover’s Distance—that neglect temporal dynamics, leading to distorted assessments. To address these issues, we propose the first utility-centric BPS evaluation paradigm, shifting away from prediction-oriented objectives and EMD-based metrics toward task-driven evaluation grounded in downstream predictive monitoring performance using simulated event logs. Our approach constructs process monitoring models, designs a transfer-learning-based comparative protocol, and decouples model accuracy from data complexity to enable attribution analysis and trustworthy diagnostics. Empirical validation across multiple real-world process datasets demonstrates that our paradigm effectively distinguishes model deficiencies from data drift, significantly enhancing the reliability and interpretability of BPS model quality assessment.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Business process simulation (BPS) is a key tool for analyzing and optimizing organizational workflows, supporting decision-making by estimating the impact of process changes. The reliability of such estimates depends on the ability of a BPS model to accurately mimic the process under analysis, making rigorous accuracy evaluation essential. However, the state-of-the-art approach to evaluating BPS models has two key limitations. First, it treats simulation as a forecasting problem, testing whether models can predict unseen future events. This fails to assess how well a model captures the as-is process, particularly when process behavior changes from train to test period. Thus, it becomes difficult to determine whether poor results stem from an inaccurate model or the inherent complexity of the data, such as unpredictable drift. Second, the evaluation approach strongly relies on Earth Mover's Distance-based metrics, which can obscure temporal patterns and thus yield misleading conclusions about simulation quality. To address these issues, we propose a novel framework that evaluates simulation quality based on its ability to generate representative process behavior. Instead of comparing simulated logs to future real-world executions, we evaluate whether predictive process monitoring models trained on simulated data perform comparably to those trained on real data for downstream analysis tasks. Empirical results show that our framework not only helps identify sources of discrepancies but also distinguishes between model accuracy and data complexity, offering a more meaningful way to assess BPS quality.

Problem

• Evaluating BPS model accuracy beyond forecasting limitations
• Addressing Earth Mover's Distance metric temporal pattern shortcomings
• Proposing utility-based framework comparing simulated and real data performance

Innovation

• Evaluates simulation quality via representative behavior generation
• Compares predictive models on simulated versus real data
• Distinguishes model accuracy from data complexity