FlowAgent: Achieving Compliance and Flexibility for Workflow Agents

2025-02-20

AI Summary

This paper addresses the challenge of balancing compliance and flexibility when integrating large language models (LLMs) with structured workflows. Methodologically, it introduces a novel LLM-based workflow agent framework featuring: (1) a Process Description Language (PDL), designed to ensure both human readability and execution precision; and (2) a controller-supervised dynamic workflow management mechanism that enables LLM-driven workflow parsing, anomaly detection, and routing of Out-of-Workflow (OOW) queries. In terms of contributions, the work establishes the first dedicated evaluation benchmark for OOW handling capability. Empirical evaluation across three datasets achieves 100% compliance on standard workflows and an average OOW response accuracy of 87.6%, substantially outperforming baseline approaches. The implementation is publicly released as open-source software.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The integration of workflows with large language models (LLMs) enables LLM-based agents to execute predefined procedures, enhancing automation in real-world applications. Traditional rule-based methods tend to limit the inherent flexibility of LLMs, as their predefined execution paths restrict the models' action space, particularly when the unexpected, out-of-workflow (OOW) queries are encountered. Conversely, prompt-based methods allow LLMs to fully control the flow, which can lead to diminished enforcement of procedural compliance. To address these challenges, we introduce FlowAgent, a novel agent framework designed to maintain both compliance and flexibility. We propose the Procedure Description Language (PDL), which combines the adaptability of natural language with the precision of code to formulate workflows. Building on PDL, we develop a comprehensive framework that empowers LLMs to manage OOW queries effectively, while keeping the execution path under the supervision of a set of controllers. Additionally, we present a new evaluation methodology to rigorously assess an LLM agent's ability to handle OOW scenarios, going beyond routine flow compliance tested in existing benchmarks. Experiments on three datasets demonstrate that FlowAgent not only adheres to workflows but also effectively manages OOW queries, highlighting its dual strengths in compliance and flexibility. The code is available at https://github.com/Lightblues/FlowAgent.

Problem

• Integrating workflows with LLMs enhances automation
• Traditional methods limit flexibility in LLM action space
• FlowAgent maintains compliance and flexibility in workflows

Innovation

• FlowAgent framework balances compliance flexibility
• Procedure Description Language merges natural language code
• New evaluation method assesses OOW query handling