Knowledge-Centric Agents for Workflow Generation

2026-07-17

AI Summary

Existing large language models often produce structurally fragile and knowledge-poor workflows when generating visual creation system pipelines (e.g., ComfyUI), primarily due to insufficient expert-level compositional reasoning capabilities. This work proposes a knowledge-centric framework that explicitly models the structure, hierarchy, and reasoning mechanisms of domain knowledge for the first time. By integrating knowledge inversion distillation, hierarchical supervised fine-tuning, and invertible reasoning, the framework enables hierarchical generation and self-optimization of executable workflows from natural language task descriptions. The approach substantially improves node diversity, structural coherence, and execution success rates of generated workflows, establishing a new foundation for knowledge-driven agent-based workflow synthesis.

Abstract

Workflow generation in visual creation systems such as ComfyUI demands not only syntactic accuracy but also expert-level reasoning over modular compositions. Existing large language model (LLM) approaches often treat this as a direct text-to-JSON generation task, struggling with structural brittleness and lacking the experiential knowledge required for effective design. We argue that successful workflow generation requires modeling knowledge itself, including its structure, hierarchy, and reasoning dynamics. To this end, we propose a knowledge-centric framework that learns to invert, inject, and infer with knowledge across multiple abstraction levels. We first perform knowledge inversion to distill hierarchical representations, ranging from full pseudo-codes and skeletons to high-level strategies, from large collections of real-world workflows. We then conduct knowledge injection through supervised fine-tuning, teaching the model to reason from task descriptions to strategies and from strategies to executable structures. During inference, the model performs reversible reasoning to synthesize executable workflows, augmented by self-refinement for structural coherence. Extensive experiments demonstrate that our method produces workflows with richer node diversity, more coherent structures, and higher execution success rates than existing systems, establishing a new foundation for knowledge-driven, agentic workflow generation.

Problem

• workflow generation
• knowledge modeling
• structural brittleness
• expert-level reasoning
• visual creation systems

Innovation

• knowledge-centric
• workflow generation
• knowledge inversion
• reversible reasoning
• hierarchical representation