Evaluating Large Language Models for Symbolic Security Protocol Analysis

2026-07-22

AI Summary

This study presents the first systematic evaluation of the feasibility of using large language models (LLMs) as substitutes for formal verification tools—such as ProVerif and OFMC—in symbolic security protocol analysis. The authors assess GPT and DeepSeek in both chat and reasoning modes on 130 obfuscated AnB/AnBx protocols, comparing their outputs against formal verification results across 388 security properties. In reasoning mode, LLMs achieve a peak F1 score of 95.7% on confidentiality tasks, yet exhibit less than 50% detection accuracy for authentication attacks. These findings highlight the potential of LLMs in confidentiality analysis while underscoring their current limitations in handling authentication properties. Although LLMs cannot yet replace formal methods, they show promise as effective pre-screening tools in protocol analysis workflows.

Abstract

Security protocol verification relies on formal tools such as ProVerif and OFMC. This study evaluates whether Large Language Models (LLMs) can perform comparable analysis. We test GPT and DeepSeek in chat and reasoning modes over three runs on 130 obfuscated AnB/AnBx protocols covering 388 security goals, scored against ProVerif and OFMC. Chat models reach 69 to 81% recall at precision below 31%. Reasoning models reverse this trade-off, reaching 66.5% precision for GPT and 45.4% for DeepSeek, but detect just over half the attacks. DeepSeek's two modes share one underlying model, so the comparison isolates reasoning itself, which raises precision from 27.2% to 45.4%. The GPT contrast spans a model-version change and is only suggestive. All models perform worst on authentication goals: reasoning models detect well under half of injective and non-injective agreement attacks, whereas chat models over-flag them at low precision. Confidentiality is the exception, with F1 up to 95.7% in reasoning mode. Verdicts are unstable across runs, identical on 89.7% of goals for GPT but 74.0% for DeepSeek. Self-reported confidence is uniformly high yet shows no meaningful correlation with correctness. On this benchmark LLMs do not match formal verification, but may serve, at best, as pre-screening filters.

Problem

• Large Language Models
• Security Protocol Analysis
• Formal Verification
• Authentication
• Confidentiality

Innovation

• Large Language Models
• Security Protocol Verification
• Symbolic Analysis
• Reasoning Mode
• Formal Methods