Technology Category
Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Computer Vision: Large Vision Models
Evaluation of LLMs for Process Model Analysis and Optimization
🤖 AI Summary
This study investigates the capability of large language models (LLMs) in analyzing and optimizing business process models (BPMN). Addressing core tasks—including BPMN diagram comprehension, syntax/logical error detection, and multi-level semantic reasoning—we propose a zero-shot, image-text collaborative natural language interaction paradigm that requires no fine-tuning or domain adaptation. Experimental results demonstrate that state-of-the-art LLMs (e.g., ChatGPT) can accurately parse BPMN diagrams and perform human-like reasoning, exhibiting strong generalization across syntactic, logical, and semantic dimensions; notable performance disparities among models further validate their viability as intelligent process design assistants. Our primary contribution is the first systematic evaluation of LLMs’ end-to-end zero-shot understanding of BPMN diagrams, establishing foundational insights into their practical utility for intelligent business process modeling and providing an empirical benchmark for future research.
Technology Category
Application Category
📝 Abstract
In this paper, we report our experience with several LLMs for their ability to understand a process model in an interactive, conversational style, find syntactical and logical errors in it, and reason with it in depth through a natural language (NL) interface. Our findings show that a vanilla, untrained LLM like ChatGPT (model o3) in a zero-shot setting is effective in understanding BPMN process models from images and answering queries about them intelligently at syntactic, logic, and semantic levels of depth. Further, different LLMs vary in performance in terms of their accuracy and effectiveness. Nevertheless, our empirical analysis shows that LLMs can play a valuable role as assistants for business process designers and users. We also study the LLM's "thought process" and ability to perform deeper reasoning in the context of process analysis and optimization. We find that the LLMs seem to exhibit anthropomorphic properties.
Problem
Research questions and friction points this paper is trying to address.
Evaluating LLMs' ability to understand process models through conversation
Assessing LLMs' effectiveness in identifying process model errors
Analyzing LLMs' reasoning capabilities for process optimization
Innovation
Methods, ideas, or system contributions that make the work stand out.
LLMs analyze BPMN models from images
LLMs detect errors in process models conversationally
LLMs assist business process design via reasoning
🔎 Similar Papers
No similar papers found.
