Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM

2026-07-30

AI Summary

This work addresses prevalent challenges in enterprise modeling tools—namely, their complexity, limited extensibility, slow feature iteration, and the low reliability of code generated by large language models. To overcome these issues, the authors propose a human-AI collaborative development framework leveraging the Claude Code AI coding agent to build PM4Py-UCM, an open-source tool that automatically derives Use Case Map (UCM) models from event logs. The approach integrates a reproducible, privacy-preserving toolkit and a classification schema to systematically identify corrective actions and consistency maintenance patterns, augmented by an Oracle-based validation strategy to enhance code trustworthiness. Empirical evaluation spans 18 AI-agent sessions, 151 commits, and 20 versions, with test functions increasing from 108 to 691. Notably, bug-fixing efforts amounted to 2.3 times the effort for new features, and approximately 18% of human interventions corrected AI-generated errors, demonstrating the method’s effectiveness and practicality.

Abstract

Enterprise-modeling (EM) tools are often complex and hard to extend. Yet, users may want to explore new EM features and capabilities that currently do not exist. AI coding agents can help here by enabling the development of new capabilities and entire tools, but whether we can trust a modeling-language tool an LLM largely wrote remains a question. This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs. PM4Py-UCM's capabilities include some expected from process mining tools (e.g., performance heat-maps and dashboards) and distinctive ones (e.g., mined executable scenarios/variants, and model decomposition). We mined the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions, in order to characterize, in a single in-depth case, how the tool was built with an agent (Claude Code), complemented by an independent static assessment of the resulting code (coverage, complexity, maintainability, security, architecture). We contribute a reproducible, privacy-preserving toolkit and taxonomy that classify human turns and flag cross-cutting consistency work, agent corrections, and retracted requests. Up to version 0.7.4, fixes outnumber features 2.3:1, with ~18% of turns for correcting agent errors. Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features. We finally present lessons learned, centered on making model transformations mechanically checkable, and the oracle-based validation strategy that closed the "the agent said it works" gap, for responsibly engineering EM tooling with AI.

Problem

• Enterprise Modeling
• Agentic AI
• Process Mining
• Tool Trustworthiness
• Use Case Map

Innovation

• Agentic AI
• Process Mining
• Use Case Maps
• AI-assisted Software Engineering
• Oracle-based Validation