Towards Leveraging LLMs to Generate Abstract Penetration Test Cases from Software Architecture

2026-03-24

AI Summary

This work addresses the critical gap in current software architecture design practices—the absence of effective security assessment mechanisms, which often allows vulnerabilities to persist into later development stages or even production. To bridge this gap, the paper introduces a novel Abstract Penetration Testing Case (APTC) metamodel that enables, for the first time, the automatic generation of security evaluation and penetration testing cases directly from software architecture models. By integrating large language models with tailored prompting strategies, the proposed approach autonomously produces APTCs with high practical utility (93%) and correctness (86%). This capability empowers architects to proactively identify security flaws at the design level and provides actionable guidance for subsequent penetration testing activities.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Metareasoning and Metaheuristics
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Software architecture models capture early design decisions that strongly influence system quality attributes, including security. However, architecture-level security assessment and feedback are often absent in practice, allowing security weaknesses to propagate into later phases of the software development lifecycle and, in some cases, to remain undiscovered, ultimately leading to vulnerable systems. In this paper, we bridge this gap by proposing the generation of Abstract Penetration Test Cases (APTCs) from software architecture models as an input to support architecture-level security assessment. We first introduce a metamodel that defines the APTC concept, and then investigate the use of large language models with different prompting strategies to generate meaningful APTCs from architecture models. To design the APTC metamodel, we analyze relevant standards and state of the art using two criteria: (i) derivability from software architecture, and (ii) usability for both architecture security assessment and subsequent penetration testing. Building on this metamodel, we then proceed to generate APTCs from software architecture models. Our evaluation shows promising results, achieving up to 93\% usefulness and 86\% correctness, indicating that the generated APTCs can substantially support both architects (by highlighting security-critical design decisions) and penetration testers (by providing actionable testing guidance).

Problem

• software architecture
• security assessment
• penetration testing
• LLMs
• abstract penetration test cases

Innovation

• Abstract Penetration Test Cases
• Software Architecture
• Large Language Models
• Security Assessment
• Metamodel