AiRacleX: Automated Detection of Price Oracle Manipulations via LLM-Driven Knowledge Mining and Prompt Generation

2025-02-10

AI Summary

Decentralized Finance (DeFi) oracles are vulnerable to price manipulation, and conventional manual auditing is inefficient and poorly scalable. Method: This paper proposes the first multi-stage large language model (LLM)-collaborative framework for oracle manipulation detection: (1) domain knowledge distillation by an LLM; (2) structured chain-of-thought (CoT) prompt generation; and (3) vulnerability pattern identification via static analysis. It integrates domain-specific knowledge distillation, CoT prompting, multi-LLM collaborative reasoning, and static analysis. Contribution/Results: The framework demonstrates, for the first time, the feasibility of open-weight LLMs in smart contract security auditing. Evaluated on 46 real-world oracle manipulation attacks, it achieves a 2.58× improvement in recall (0.667 vs. 0.259) over prior work while matching GPTScan’s precision. The optimal configuration—Haiku-Haiku-4o-mini—balances detection performance, privacy preservation, and deployment cost.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Decentralized finance applications depend on accurate price oracles to ensure secure transactions, yet these oracles are highly vulnerable to manipulation, enabling attackers to exploit smart contract vulnerabilities for unfair asset valuation and financial gain. Detecting such manipulations traditionally relies on the manual effort of experienced experts, presenting significant challenges. In this paper, we propose a novel LLM-driven framework that automates the detection of price oracle manipulations by leveraging the complementary strengths of different LLM models. Our approach begins with domain-specific knowledge extraction, where an LLM model synthesizes precise insights about price oracle vulnerabilities from top-tier academic papers, eliminating the need for profound expertise from developers or auditors. This knowledge forms the foundation for a second LLM model to generate structured, context-aware chain of thought prompts, which guide a third LLM model in accurately identifying manipulation patterns in smart contracts. We validate the framework effectiveness through experiments on 60 known vulnerabilities from 46 real-world DeFi attacks or projects spanning 2021 to 2023. The best performing combination of LLMs (Haiku-Haiku-4o-mini) identified by AiRacleX demonstrate a 2.58-times improvement in recall (0.667 vs 0.259) compared to the state-of-the-art tool GPTScan, while maintaining comparable precision. Furthermore, our framework demonstrates the feasibility of replacing commercial models with open-source alternatives, enhancing privacy and security for developers.

Problem

• Automated detection of price oracle manipulations
• LLM-driven knowledge mining and prompt generation
• Enhancing security in decentralized finance applications

Innovation

• LLM-driven knowledge extraction
• Context-aware prompt generation
• Open-source model replacement feasibility