Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform

2026-08-04

AI Summary

This study addresses the urgent need for automated workflows in enterprise compliance management within hybrid cloud environments, where sub-second responsiveness and structural correctness are critical—requirements that traditional static orchestrators struggle to meet. To this end, the authors propose a chunked workflow generation pipeline that decomposes workflow construction into three stages: variable scaffolding, atomic block assembly, and nested block generation, substantially improving structural validity. Evaluated across 29 real-world scenarios and 2,784 experiments using six large language models—including variants from the Mistral series and GPT-OSS-120B—the approach demonstrates a significant trade-off optimization among cost, latency, and success rate. Structural success rates improved from 31.5%–82.8% to 74.1%–97.8%, with Mistral-small achieving a 95.7% success rate at just $0.01 per workflow, meeting both interactive latency constraints (<50 seconds) and production deployment standards.

Abstract

Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.

Problem

• enterprise automation
• workflow generation
• compliance management
• large language models
• hybrid cloud

Innovation

• piecewise pipeline
• workflow generation
• structural validity
• LLM cost-latency trade-off
• enterprise automation