Generative Large Language Model usage in Smart Contract Vulnerability Detection

2025-04-07

AI Summary

This study systematically evaluates the effectiveness of large language models (LLMs) in smart contract vulnerability detection, benchmarking against state-of-the-art static (Slither) and dynamic (Mythril) analyzers. To address the lack of realistic, vulnerability-grounded evaluation, we propose the first joint LLM–traditional-tool benchmarking framework tailored to real-world contract vulnerabilities. Our method introduces a lightweight, vulnerability-aware pretraining strategy for LLMs and designs a hybrid detection architecture that synergistically integrates semantic understanding with precise control-flow and path-sensitive analysis. Experimental results show that while current LLMs cannot replace traditional tools, they excel at semantically complex vulnerabilities—e.g., reentrancy and logic errors—yet underperform on path- and arithmetic-sensitive flaws such as integer overflows and access control violations. Our key contributions are: (1) empirically establishing the capability boundaries of LLMs in smart contract security analysis; (2) proposing a scalable hybrid paradigm; and (3) introducing a lightweight adaptation pathway for domain-specific LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent years have seen an explosion of activity in Generative AI, specifically Large Language Models (LLMs), revolutionising applications across various fields. Smart contract vulnerability detection is no exception; as smart contracts exist on public chains and can have billions of dollars transacted daily, continuous improvement in vulnerability detection is crucial. This has led to many researchers investigating the usage of generative large language models (LLMs) to aid in detecting vulnerabilities in smart contracts. This paper presents a systematic review of the current LLM-based smart contract vulnerability detection tools, comparing them against traditional static and dynamic analysis tools Slither and Mythril. Our analysis highlights key areas where each performs better and shows that while these tools show promise, the LLM-based tools available for testing are not ready to replace more traditional tools. We conclude with recommendations on how LLMs are best used in the vulnerability detection process and offer insights for improving on the state-of-the-art via hybrid approaches and targeted pre-training of much smaller models.

Problem

• Evaluating LLM-based tools for smart contract vulnerability detection
• Comparing LLM tools with traditional static and dynamic analysis methods
• Proposing hybrid approaches to improve vulnerability detection accuracy

Innovation

• LLMs for smart contract vulnerability detection
• Comparison with traditional static dynamic tools
• Hybrid approaches and targeted pre-training