R+R: Reassessing Java Security API Misuse in Current LLMs: A Replication on JCA and JSSE APIs with External Security Knowledge

2026-05-29

AI Summary

This study addresses the widespread misuse of Java Cryptographic Architecture (JCA) and Java Secure Socket Extension (JSSE) APIs in code generated by large language models, which poses significant security risks to software systems. It presents the first systematic evaluation of GPT-5.5 and Llama-3.3-70B-Instruct on this task and investigates the impact of external security knowledge—including secure coding examples, known misuse patterns, and developer guidelines—on mitigating such vulnerabilities. Through benchmarking and security-oriented prompt engineering, the findings reveal that while newer models exhibit overall improvements, API misuses persist. Crucially, integrating external knowledge substantially enhances code safety; notably, GPT-5.5 completely eliminates security misuses in valid programs when explicitly provided with misuse patterns, highlighting a synergistic effect between model capabilities and targeted knowledge injection strategies.

Abstract

The misuse of Java security APIs is a serious security problem in software development. Research in 2024 has shown that this problem is widespread in LLM-generated code. However, it remains unclear whether this phenomenon persists in current models and how external security knowledge affects it. This paper presents a scoped replication and extension of Mousavi et al.'s study on the Java Cryptography Architecture (JCA) and Java Secure Socket Extension (JSSE) APIs. We focus on two complementary settings: GPT-5.5 as a frontier proprietary coding model, and Llama-3.3-70B-Instruct as a strong open-weight model relevant to self-hosted deployment. The results show that although newer LLMs perform better in using Java security APIs, the problem of Java security API misuse has not been eliminated. External security knowledge substantially improves the measured outcome, but its effect is model-dependent. For Llama-3.3-70B-Instruct, secure code examples are the most effective single knowledge type. For GPT-5.5, explicit misuse patterns eliminate all detected security API misuses among valid programs in our benchmark, although some outputs remain invalid due to compilation errors or target-API mismatches. In addition, developer-guide knowledge becomes much more effective, and secure prompting also provides large gains for GPT-5.5. Overall, these findings confirm the Java security API misuse risk identified in the original study and show that the benefits of retrieval-augmented knowledge depend not only on the knowledge itself and retrieval behavior, but also on model capability.

Problem

• Java security API misuse
• LLM-generated code
• JCA
• JSSE
• security knowledge

Innovation

• LLM security
• Java security API misuse
• retrieval-augmented generation
• model-dependent knowledge effectiveness
• secure code generation