toolchain automation

This study addresses the challenges of automatically reconstructing BPMN models from unstructured natural language descriptions, including specification heterogeneity, multilingual inputs, and the absence of ground-truth references. To overcome these issues, the authors propose a large language model (LLM)-driven, multi-stage automation pipeline that integrates multilingual translation, SpiffWorkflow-based execution validation, and LLM-guided iterative repair to generate high-quality, executable BPMN 2.0 XML ground-truth corpora. A novel multidimensional similarity evaluation framework—combining structural alignment, type distribution, and semantic embeddings—is introduced to enable fully automated, large-scale BPMN generation and refinement without manual intervention. Evaluated on 750 public process diagrams, the approach successfully constructs 387 validated models with an average reconstruction similarity exceeding 0.75, including approximately 50 near-perfect reconstructions differing only in element naming.

This work investigates the robustness of large language models (LLMs) in automated formalization of mathematical statements, specifically their stability under semantically equivalent yet syntactically distinct natural language inputs. Method: We introduce, for the first time in this domain, systematic semantic-preserving rewrites of problem statements and evaluate LLMs on the MiniF2F and Lean 4–based ProofNet benchmarks. Two state-of-the-art LLMs generate formal proofs, with outputs cross-validated for semantic consistency and compilation validity. Contribution/Results: Despite high semantic equivalence among inputs, minor paraphrasing induces substantial variation in formalized outputs—revealing acute sensitivity to surface-level linguistic variations. Our findings expose a critical robustness gap in current automated formalization systems and establish a reproducible evaluation framework grounded in empirical evidence, providing concrete foundations for improving model stability in formal reasoning tasks.

This work addresses the lack of formal fidelity verification methods for natural language outputs—such as Gherkin scenarios—generated by large language models (LLMs). To this end, we propose a logic-based consistency verification framework grounded in automated formalization. Methodologically, we introduce automated formalization to LLM output validation for the first time: an LLM-driven formalizer translates both natural language requirements and LLM-generated outputs into first-order logic formulas; formal reasoning is then applied to assess semantic equivalence and detect logical contradictions. Experiments demonstrate that our approach effectively identifies semantic equivalence across paraphrased expressions and uncovers latent logical inconsistencies, thereby significantly enhancing the trustworthiness of generated artifacts. Our primary contribution is establishing the first formal verification paradigm tailored to LLM-generated outputs, providing both theoretical foundations and practical methodology for ensuring the verifiability of automated artifacts in requirements engineering.

Large language models (LLMs) exhibit limited performance on scientific workflow tasks—including configuration, annotation, translation, explanation, and generation—primarily due to insufficient domain knowledge. Method: This work presents the first systematic evaluation of over 20 open- and closed-source LLMs (e.g., Llama, GPT series) across mainstream workflow systems (e.g., Apache Airflow, Snakemake), employing customized prompts and a multidimensional evaluation protocol tailored to workflow semantics and execution constraints. Results: LLM accuracy on workflow tasks is substantially lower than on general NLP benchmarks; cross-system performance varies by over 40%, confirming that capabilities are highly sensitive to both task type and system architecture. The study identifies domain knowledge deficiency as the fundamental bottleneck and proposes transferable prompt optimization strategies and domain alignment techniques. It establishes the first empirical benchmark and methodological framework for leveraging LLMs in research automation.

Large language models (LLMs) struggle with autonomous understanding and invocation of external tools. Method: This paper proposes AutoTools, an end-to-end automatic tool learning framework, and AutoTools-learning, a corresponding training paradigm. It eliminates reliance on human demonstrations, special tokens, or hard-coded integrations, instead leveraging LLM-driven tool documentation parsing, function synthesis, and program generation to achieve zero-shot tool adaptation and dynamic multi-tool orchestration. Contribution/Results: Training proceeds in three synthetic-data stages—documentation understanding, relevance learning, and function programming—yielding substantial gains on a newly constructed high-difficulty benchmark. With only 34K synthetic samples, open-source small models achieve up to a 41.2% absolute improvement in tool-call accuracy, marking the first demonstration of efficient generalization for compact models on complex tool-use tasks.

This work proposes the first fully large language model–driven, end-to-end data integration framework that eliminates the need for manual configuration, which traditionally incurs high costs and low efficiency. The system autonomously generates a complete integration pipeline encompassing schema mapping, value normalization, entity matching, and conflict resolution without human intervention. Evaluated on three real-world domains—gaming, music, and enterprise data—the GPT-5.2–based framework achieves integration performance comparable to or surpassing that of handcrafted systems. Notably, it accomplishes this at a remarkably low cost of approximately $10 per execution, substantially reducing human labor and operational overhead.

This study addresses the frequent failure of large language models (LLMs) in software engineering tasks due to structurally invalid outputs—such as syntactic or formatting errors—that prevent correct parsing by downstream toolchains, even when the semantic content is accurate. The authors systematically evaluate the reliability of structured output generation across four representative tasks, categorizing errors into syntactic, structural, and semantic types. They propose a template-driven token-matching generation (TTMG) method that enforces structural consistency during autoregressive decoding. Experimental results demonstrate that TTMG nearly eliminates syntactic errors; however, structural and semantic errors remain prevalent. This work reveals, for the first time, that the fundamental bottleneck in structured output generation lies not merely in syntax but in the insufficient coordination between structural and semantic correctness, indicating that existing structural control mechanisms, while necessary, are insufficient without joint guarantees of both dimensions.

This work addresses the high latency of multi-step tool calling, which severely hinders the deployment of large language models in real-time services. To this end, it introduces a training-free, plug-and-play acceleration method that integrates structured tool-calling patterns and retrieval-augmented mechanisms into a speculative decoding framework for the first time. The approach employs a finite-state machine to alternately fill pattern tokens and speculatively generate variable fields, while leveraging vector retrieval to reuse historical tool-call records as drafts, substantially improving generation efficiency. Experimental results demonstrate that the proposed method achieves up to 4.2× inference speedup across multiple benchmarks, significantly outperforming existing training-free speculative decoding strategies.

This work addresses the prevalent issue in large language models (LLMs) of introducing control-flow, type, or I/O errors during code translation due to neglect of program intent. To mitigate this, the paper proposes the first systematic use of a language-agnostic, structured intermediate specification that preserves semantic fidelity through an intermediate representation, structured generation, and automated test-based validation. Evaluated on the Avatar and CodeNet datasets with five state-of-the-art LLMs, the approach significantly improves translation accuracy, raising the micro-averaged accuracy from 67.7% to 78.5%. It completely eliminates lexical errors and substantially reduces errors related to structure, declarations, and runtime dependencies.

The era of large language models (LLMs) faces critical challenges including insufficient high-quality data supply, fragmented data preparation pipelines, poor reproducibility, and lack of model-in-the-loop support. Method: We propose the first LLM-driven, unified data preparation framework for data-centric AI, featuring system-level abstractions and PyTorch-style APIs for modular design. We introduce DataFlow-Agent—the first agent that synthesizes executable data pipelines end-to-end from natural language specifications—and integrate LLM-powered operator synthesis, iterative validation, 200+ reusable operators, and six domain-agnostic pipeline templates. Results: Experiments on Text-to-SQL, code generation, and mathematical reasoning show our synthesized data significantly outperforms human-annotated and domain-specific synthetic data. Remarkably, just 10K samples surpass the performance of models trained on the million-scale Infinity-Instruct dataset, empirically validating the decisive impact of data quality on model performance.