CURVALID: Geometrically-guided Adversarial Prompt Detection

2025-03-05

AI Summary

Large language models (LLMs) are vulnerable to adversarial prompt attacks that induce jailbreaking and harmful behaviors. Method: This paper proposes the first geometry-based universal prompt detection framework, requiring no model modification or architecture-specific assumptions. It generalizes the concept of curvature to *n*-dimensional token embedding spaces via the Whewell equation and integrates local intrinsic dimensionality (LID) to characterize geometric anomalies of prompts within adversarial subspaces. By jointly modeling curvature and local dimensional properties of semantic manifolds, the method captures structural deviations induced by adversarial perturbations. Contribution/Results: The framework achieves high-accuracy, cross-model, and cross-attack detection of adversarial prompts. Extensive experiments on mainstream LLMs demonstrate significant improvements over existing defenses. It establishes a transferable, interpretable geometric analysis paradigm for secure LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Adversarial prompts capable of jailbreaking large language models (LLMs) and inducing undesirable behaviours pose a significant obstacle to their safe deployment. Current mitigation strategies rely on activating built-in defence mechanisms or fine-tuning the LLMs, but the fundamental distinctions between adversarial and benign prompts are yet to be understood. In this work, we introduce CurvaLID, a novel defense framework that efficiently detects adversarial prompts by leveraging their geometric properties. It is agnostic to the type of LLM, offering a unified detection framework across diverse adversarial prompts and LLM architectures. CurvaLID builds on the geometric analysis of text prompts to uncover their underlying differences. We theoretically extend the concept of curvature via the Whewell equation into an $n$-dimensional word embedding space, enabling us to quantify local geometric properties, including semantic shifts and curvature in the underlying manifolds. Additionally, we employ Local Intrinsic Dimensionality (LID) to capture geometric features of text prompts within adversarial subspaces. Our findings reveal that adversarial prompts differ fundamentally from benign prompts in terms of their geometric characteristics. Our results demonstrate that CurvaLID delivers superior detection and rejection of adversarial queries, paving the way for safer LLM deployment. The source code can be found at https://github.com/Cancanxxx/CurvaLID

Problem

• Detects adversarial prompts in large language models
• Leverages geometric properties for prompt analysis
• Provides a unified framework across diverse LLM architectures

Innovation

• Leverages geometric properties for adversarial prompt detection
• Extends curvature concept to n-dimensional word embeddings
• Uses Local Intrinsic Dimensionality for geometric feature analysis