Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation

2025-05-28

AI Summary

This paper addresses anomaly detection in process execution without predefined process models. To this end, it introduces retrieval-augmented generation (RAG) into process compliance checking for the first time, proposing a modeling-free and fine-tuning-free large language model (LLM)-driven approach. The method constructs a cross-process behavioral knowledge base via event log analysis and frequent trajectory mining; RAG then enables the LLM to dynamically retrieve positive and negative behavioral exemplars, facilitating context-aware knowledge transfer and zero-model-dependency anomaly identification. Experiments on multiple real-world event logs demonstrate that the proposed method achieves an average 12.7% improvement in F1-score over fine-tuned LLM baselines. These results validate RAG as a lightweight, efficient, and generalizable new paradigm for process compliance checking.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Conformance checking techniques detect undesired process behavior by comparing process executions that are recorded in event logs to desired behavior that is captured in a dedicated process model. If such models are not available, conformance checking techniques are not applicable, but organizations might still be interested in detecting undesired behavior in their processes. To enable this, existing approaches use Large Language Models (LLMs), assuming that they can learn to distinguish desired from undesired behavior through fine-tuning. However, fine-tuning is highly resource-intensive and the fine-tuned LLMs often do not generalize well. To address these limitations, we propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior. Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base that contains both desired and undesired process behavior from other processes, assuming that the LLM can transfer this knowledge to the process at hand. Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior, demonstrating that RAG is a viable alternative to resource-intensive fine-tuning, particularly when enriched with relevant context from the event log, such as frequent traces and activities.

Problem

• Detect undesired process behavior without dedicated process models
• Overcome limitations of resource-intensive LLM fine-tuning
• Utilize Retrieval Augmented Generation for knowledge transfer

Innovation

• Uses Retrieval Augmented Generation (RAG)
• Eliminates need for dedicated process models
• Avoids resource-intensive LLM fine-tuning