causal workflow orchestration

This work addresses the heavy reliance on expert knowledge in designing and debugging scientific workflows, a challenge exacerbated by existing large language model approaches that directly generate code without ensuring transparency, reproducibility, or seamless system integration. To overcome these limitations, we propose an AI-assisted scientific workflow management framework that decouples user intent from implementation through a structured specification phase, enabling specification-driven workflow generation and validation. We further introduce a multi-layer debugging agent powered by large language models to automate error diagnosis and correction. By deeply integrating with the Pegasus workflow system via the Model Context Protocol (MCP), our approach supports end-to-end workflow lifecycle management. Empirical evaluation demonstrates successful generation and execution of federated learning medical imaging workflows comprising thousands of tasks, substantially reducing debugging effort and empowering non-expert users to construct complex workflows adhering to expert-level design patterns.

To address the challenges of parallel scheduling, opaque execution states, poor result reproducibility, and inadequate auditability when managing hundreds to thousands of Snakemake/Nextflow pipelines in large-scale bioinformatics analyses, this paper proposes a lightweight command-line orchestration framework. Built in Python and integrated with SQLite or PostgreSQL, it enables unified pipeline launching across heterogeneous workflows, real-time status monitoring, fine-grained log collection, automated result ingestion into databases, and comprehensive lifecycle metric logging—including runtime, resource consumption, and failure points. It introduces a novel CLI paradigm that supports cross-pipeline collaborative monitoring and reproducibility assurance without modifying existing workflow code. Experimental evaluation demonstrates a 42% improvement in multi-project throughput, significantly enhancing observability, auditability, and reproducibility in large-scale bioinformatics analysis.

This work addresses the unreliability of developer productivity dashboards, which often stems from ad hoc scripts that introduce undetected silent data gaps, eroding organizational trust. To resolve this, we propose a robust ELT pipeline grounded in DAG-based orchestration and the Medallion architecture, decoupling data extraction from transformation to preserve the immutability of raw data. Our approach introduces a state-driven dependency scheduling mechanism and, for the first time, treats metric pipelines as production-grade distributed systems. We emphasize the critical role of immutable raw history in enabling reliable metric redefinition. This methodology significantly enhances data reliability and freshness while effectively eliminating silent failures, thereby restoring organizational confidence in DevOps metrics.

To address the challenge of efficiently orchestrating and intelligently managing complex, dynamic workflows in large-scale distributed scientific computing, this paper proposes an integrated intelligent workflow system that unifies task scheduling, data movement, and adaptive decision-making. The system supports data-aware execution, conditional logic, and programmable directed acyclic graphs (DAGs), operating in both template-driven and “function-as-a-task” modes. It adopts a modular, message-driven architecture and deeply integrates mainstream middleware—including PanDA and Rucio—while incorporating distributed hyperparameter optimization and AI-assisted modeling. Its cross-experiment, cross-platform design significantly enhances scalability and reproducibility. Deployed in major scientific projects—including ATLAS, the Rubin Observatory, and the Electron-Ion Collider—the system enables high-throughput execution of heterogeneous tasks and reduces operational overhead by over 30%.

This work addresses the limitations of traditional high-performance computing (HPC), which relies on manual task scripting and scheduling and struggles to meet the automation demands of complex scientific workflows. The authors propose the first large language model–based autonomous agent framework that enables end-to-end automated execution of HPC workflows from descriptive instructions. The framework integrates Slurm/Flux job schedulers, low-latency AWS cloud infrastructure, and event monitoring mechanisms to support task definition, optimization, and scheduling. Experimental results demonstrate that the system efficiently deploys scalable experiments, accurately translates job specifications—with only occasional deviations in processor affinity—and successfully reproduces an expert-level variant calling pipeline, achieving consistent results in 18 out of 19 runs. These findings validate the framework’s feasibility and effectiveness in real-world HPC environments.

Enterprise operational workflows are notoriously difficult to automate end-to-end due to their heavy reliance on human intervention and limited adaptability to change. This work proposes the first action-centric workflow graph framework, which achieves automated construction, execution, and evolution through a three-stage pipeline: structured workflow graphs are extracted from human operation traces, executed via multi-agent online traversal, and continuously optimized in a closed loop using an Adaptive Traversal Reinforcement (ATR) mechanism. Integrating large-scale offline graph construction, graph-guided retrieval, and large language model reasoning, the approach was deployed across four cloud database services. It substantially outperforms the Trace-RAG baseline in coverage breadth, factual accuracy, and diagnostic throughput, achieving an expert blind-review score of 4.95 out of 5.

Current evaluations of bioinformatics agents overemphasize answer correctness while neglecting workflow auditability and scientific credibility. This work proposes a Function–Evidence–Validation (FEV) tri-dimensional evaluation framework centered on inspectable workflow trajectories, shifting the primary focus to workflow correctness for the first time. Through systematic literature review, trajectory analysis, and cross-domain benchmark mapping, the study comprehensively analyzes 109 agent systems and 28 evaluation resources across subfields including genomics, single-cell and spatial omics, and protein science. The findings reveal that while agents perform adequately in planning and execution, they exhibit significant deficiencies in reproducibility, traceability, external validation, and prospective experimental design. This research provides both theoretical grounding and practical guidance for developing transparent, auditable next-generation bioinformatics agents.

This work addresses the limitation of existing benchmarks, which focus solely on accuracy in multi-agent orchestration tasks while neglecting fine-grained diagnosis of failure origins and recovery capabilities. The authors propose a reproducible fault-injection framework to systematically evaluate failure modes, task decomposition quality, and recovery mechanisms within templated enterprise workflows. They introduce two novel metrics: “cascade radius” and failure-mode-specific recovery rates, and employ controlled probes to analyze recovery behavior across different fault types. Experimental results demonstrate that intent-based reasoning routing achieves 100% recovery under adversarial conditions, significantly outperforming keyword-based routing; tool-related failures are fully recoverable, whereas semantic failures prove largely irrecoverable; and cascade radius increases with workflow depth.