From Backlog Items to Security Guidance: Towards Continuous Security Compliance

2026-07-29

AI Summary

This work addresses the challenge engineering teams face in explicitly identifying security requirements from unstructured to-do items in regulated domains. The authors propose a natural language processing–based to-do enhancement system that integrates a high-recall security relevance classifier with a four-stage retrieval-augmented generation (RAG) pipeline to automatically detect security-related entries and link them to compliance requirements. The study introduces the first publicly released dataset of security to-do items annotated by domain experts. The classifier achieves an in-distribution F2 score of 0.774 and demonstrates robust generalization, attaining an average zero-shot G-measure of approximately 0.65 across five benchmarks. In expert evaluation, 12 out of 24 regulatory clauses retrieved by the RAG pipeline received scores of at least 4 out of 5, confirming the method’s practical feasibility in industrial settings.

Abstract

Continuous software engineering in regulated domains requires engineering teams to address security throughout the development lifecycle. Yet making security requirements explicit in backlog items is still problematic. Engineers must instead infer security relevance of backlog items from brief, free-form descriptions and often lack timely guidance on applicable requirements. We present an NLP-based backlog enrichment system that detects security-relevant backlog items and links them to relevant security requirements. The approach combines a security-relevance classifier with a retrieval-augmented generation (RAG) pipeline over security requirements documents. The approach was developed and evaluated in the context of a large enterprise in highly regulated domains. We present three contributions. First, we release a dataset of 288 backlog items labeled for security relevance by nine security practitioners, with substantial agreement (Fleiss' $κ=0.787$). Second, a recall-oriented classifier achieving $F2=0.774$ in-distribution and mean zero-shot G-measure $\approx 0.65$ across five established benchmarks, matching or outperforming most published classical-ML and open-source GPT baselines. Third, we preliminarily evaluated a four-stage security requirements document-grounded RAG pipeline with two practitioners on industrial backlogs using company-internal security policies and CIS Benchmarks. Of the retrieved 24 clauses, 12 were rated at least 4/5 for relevance. Our findings provide first indicators that NLP-based product backlog enrichment can support engineers in identifying security requirements early in the development process. With this work we aim to facilitate continuous security compliance through proactive introduction of security requirements in continuous software engineering.

Problem

• security compliance
• backlog items
• continuous software engineering
• security requirements
• regulated domains

Innovation

• NLP-based backlog enrichment
• security-relevance classification
• retrieval-augmented generation (RAG)
• continuous security compliance
• software engineering in regulated domains