EntailLLM: Verifying LLM-Generated Vulnerability Discovery Paths with Domain Knowledge via Logic Programming

2026-08-03

AI Summary

This work addresses the unreliability of large language models (LLMs) in software vulnerability analysis, where they often produce reasoning paths inconsistent with domain knowledge—posing significant risks to safety-critical systems such as medical devices. To mitigate this, we propose the first end-to-end framework that integrates formal logical entailment into LLM-based vulnerability reasoning. Our approach employs temporally annotated logic programming to align LLM-generated vulnerability paths with an independent domain knowledge graph and leverages function call graph traversal for semantically rigorous, interpretable verification. Experimental evaluation across three CWE categories, four LLMs, and seven binary programs demonstrates that our method improves overall entailment accuracy from 78% to 98%, with performance degradation in only 3% of cases, and has been successfully deployed on real-world medical device binaries.

Abstract

Large language models are increasingly used to reason about software vulnerabilities, but their outputs can silently violate domain knowledge, limiting their reliability in safety-critical settings such as medical devices. Prior work either treats that output as a prediction to be scored or constrains it to walks within a single knowledge graph; neither checks whether reasoning over a binary is consistent with an independent body of domain knowledge. We present EntailLLM, which validates each LLM-proposed analyst path by entailment: the path is a traversal of the binary's function call graph, the domain knowledge is represented in a separate graph, and verification aligns the two under temporal annotated logic. Across three CWE classes, four LLMs, three prompting strategies, and seven binaries varying in size from 405 to 12,696 function call-graph nodes, domain knowledge raises pooled entailment from 78% to 98%, with entailment decreasing in only 3% of the experiments. EntailLLM is deployed end-to-end on real medical-device binaries, reaching 98% pooled entailment without per-device tuning. Our system inherits the formal guarantees of generalized annotated logic, providing logical verification of LLM output that is both explainable and grounded in well-defined semantics.

Problem

• LLM reliability
• software vulnerability
• domain knowledge
• logic verification
• safety-critical systems

Innovation

• EntailLLM
• logic programming
• domain knowledge
• temporal annotated logic
• vulnerability discovery