llm integration

To address high manual effort, delayed response, and incomplete coverage in industrial software test maintenance, this paper proposes and empirically validates two novel multi-agent architectures leveraging large language models (LLMs) to accurately identify test cases requiring maintenance—and to assist in their automated addition, deletion, or modification—following code changes. The study systematically distills critical triggering conditions and practical deployment constraints for LLMs in industrial settings, and conducts empirical evaluation using real-world data from Ericsson AB. Results demonstrate significant reductions in manual test maintenance effort, improved timeliness of issue response, and enhanced test coverage completeness. This work represents the first application of multi-agent collaboration to industrial-scale test maintenance, establishing a reusable methodological framework and empirical foundation for deploying LLMs in high-reliability software engineering contexts.

There is a lack of systematic, software engineering (SE)-driven analysis of the full lifecycle challenges associated with large language models (LLMs). Method: We conduct a structured systematic literature review grounded in the SE lifecycle model, decomposing LLM development into six phases—requirements, data, development, testing, deployment, and maintenance—and perform stage-wise problem identification. Contribution/Results: We uncover phase-specific SE challenges—including prompt engineering maintainability, insufficient test coverage under data drift, and misalignment between model versioning and code evolution—and propose the first SE-oriented research framework for LLMs. This framework delineates stage-specific research directions and integrated technical pathways. Our work bridges a critical theoretical gap at the intersection of LLMs and SE, delivering an actionable research roadmap and practical guidance for building efficient, reliable, and evolvable LLM-based software systems.

To address the inefficiency and poor generalization of traditional search strategies in API synthesis under large-scale implementation spaces, this paper introduces, for the first time, a systematic large language model (LLM)-driven approach to component-based API synthesis. We propose an end-to-end method leveraging prompt engineering, in-context learning, and code generation to replace exhaustive search. Evaluated on a realistic programming task benchmark comprising 135 tasks, our method successfully synthesizes correct APIs for 133 tasks—achieving a 98.5% accuracy rate—significantly outperforming the state-of-the-art tool FrAngel. Our key contributions are: (1) establishing the first LLM-driven paradigm specifically designed for API synthesis; (2) empirically demonstrating LLMs’ capability to jointly model developer intent and code patterns; and (3) substantially improving both synthesis efficiency and cross-task generalization performance.

To address ambiguity, hallucination, and logical inconsistencies arising from excessive reliance on large language models (LLMs) in modeling and simulation (M&S), this paper proposes a novel middleware paradigm wherein the LLM serves solely as a “semantic translator” rather than a decision-making agent, enabling high-fidelity interoperability across heterogeneous domain-specific tools. Methodologically, we integrate LoRA-based lightweight fine-tuning with M&S-oriented tool selection criteria to preserve tool autonomy while ensuring context-aware LLM assistance. We further design a semantic mapping middleware and structured API to support cross-tool command parsing and code generation across diverse formal modeling frameworks. Experimental results demonstrate a >30% reduction in modeling entry barriers, zero performance bottlenecks in typical M&S pipelines, and a 72% error-rate reduction compared to end-to-end LLM approaches.

Evaluating LLM-based agents is challenging due to their dynamic, probabilistic, and continuously evolving nature—traditional predefined benchmarks fail to capture open-ended behaviors, emergent outcomes, and lifecycle adaptation. Method: We propose an evaluation-driven agent development paradigm, integrating online runtime evaluation with offline reconstruction evaluation. Our hybrid framework enables real-time feedback injection, human-AI collaborative closed-loop refinement, and iterative optimization across the full stack (pipeline, architecture, and LLM), incorporating both human and AI evaluators. Contribution/Results: We introduce the first evaluation-centric process model and reference architecture for LLM agent development. It uniquely supports open-behavior capture, emergent-result governance, and dynamic alignment. Experiments demonstrate that the framework effectively enables safe, controllable, and continuous agent iteration under objective drift, requirement changes, and regulatory evolution—achieving robust adaptability without compromising reliability or compliance.

This study investigates the feasibility of leveraging low-cost, deployable open-source large language models (ranging from 0.5B to 32B parameters) to automatically generate domain-specific language (DSL) representations of UI and data models directly from natural language prompts, without fine-tuning and using only few-shot prompting. It presents the first systematic evaluation of small-scale open-source models on the task of generating multiple, interrelated DSL artifacts. Through a combination of DSL grammar parsing, automated validation, and expert assessment, the work examines model performance in terms of syntactic correctness, semantic completeness, and cross-model referential consistency. Experimental results demonstrate that compact models—such as gemma3:12b and mistral:7b-instruct—achieve generation quality comparable to or even rivaling that of significantly larger models, highlighting their practical viability and cost-effectiveness for model-driven engineering applications.

This study addresses the challenges students face when spontaneously using large language models (LLMs) in software testing education, where inadequate prompting and inefficient interactions—stemming from a lack of systematic guidance—undermine both testing behaviors and learning outcomes. Employing a mixed-methods approach that integrates analysis of classroom artifacts, competition surveys, and instructional interventions, the research identifies recurrent interaction breakdowns, including missing context, insufficient constraints, and rigid single-turn prompts. In response, it proposes the first stage-aware prompting scaffold tailored to testing tasks, which guides students to structurally articulate environmental assumptions, interaction rationale, synchronization mechanisms, and verification intent. Empirical results demonstrate that this scaffold significantly enhances the quality of student prompts and the effectiveness of generated test scripts.

Current evaluation methods for large language model (LLM) agents are fragmented and often divorced from real-world software development contexts, limiting their ability to accurately assess practical engineering capabilities. This work proposes a comprehensive evaluation framework grounded in authentic development practices, integrating for the first time a contamination-aware mechanism, real-environment agent behavior tracing, and trajectory-aware benchmark metrics. The framework systematically evaluates model performance across key dimensions including contextual understanding of code, alignment with human developer behaviors, and failure mode analysis. By bridging the gap between synthetic benchmarks and actual development workflows, this approach substantially enhances both the validity and practical utility of LLM agent evaluations, establishing a more reliable and developer-relevant assessment paradigm.

This work addresses the challenge of reliably translating natural language into industrial-grade, deployable SysMLv2 models. The authors propose an iterative generate-check-repair framework that, for the first time, integrates a production-level SysMLv2 conformance checker directly into the generation process as a control mechanism rather than a post-processing step. By combining large language model (LLM) generation with deterministic diagnostic feedback and targeted repair strategies—and terminating only when zero errors remain—the method achieves perfect compliance. Evaluated across 604 test cases derived from 151 prompts and four distinct LLMs, the approach elevates single-pass generation compliance from 51.16% to 100%, enabling robust, direct translation of natural language specifications into engineering-ready SysMLv2 models.

This work addresses the lack of structured, verifiable, and governable tool support for large language model (LLM) agents in operational tasks, where existing approaches are often static or manually integrated, struggling to balance security and extensibility. The authors propose the “Tool Capsule” paradigm, which encapsulates tools as standardized units comprising intent, contract, implementation, policy, and verification evidence. They design an efficient intent-scoped routing mechanism enabling on-demand, secure tool invocation. The system integrates a sandboxed verification pipeline, MCP-compatible routing, credential binding, and lifecycle governance. Experiments demonstrate a micro F1 score of 0.901 across 83 routing tests with a 99.2% reduction in context overhead; all 25 end-to-end tasks produced valid toolkits (micro F1 = 0.940), with 23 successfully passing real-time sandbox validation.