Peering Behind the Shield: Guardrail Identification in Large Language Models

2025-02-03

AI Summary

This work addresses the challenge of detecting and verifying implicitly deployed input/output safety guardrails in large language models (LLMs), which are often inaccessible to auditors. We propose AP-Test, the first transferable black-box framework for guardrail identification—requiring no model access or internal knowledge. AP-Test constructs guardrail-specific adversarial prompts to infer guardrail presence and incorporates an interpretable loss term to enable fine-grained attribution and red-teaming support. Technically, it integrates adversarial prompt engineering, black-box system identification, and loss-driven guardrail response modeling. Evaluated across four major guardrail categories, AP-Test achieves a mean identification accuracy of 92.7%, substantially outperforming baselines. Ablation studies confirm the critical contributions of each component. Our approach establishes a novel, scalable, interpretable, and white-box-free paradigm for LLM safety assessment.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Human-AI conversations have gained increasing attention since the era of large language models. Consequently, more techniques, such as input/output guardrails and safety alignment, are proposed to prevent potential misuse of such Human-AI conversations. However, the ability to identify these guardrails has significant implications, both for adversarial exploitation and for auditing purposes by red team operators. In this work, we propose a novel method, AP-Test, which identifies the presence of a candidate guardrail by leveraging guardrail-specific adversarial prompts to query the AI agent. Extensive experiments of four candidate guardrails under diverse scenarios showcase the effectiveness of our method. The ablation study further illustrates the importance of the components we designed, such as the loss terms.

Problem

• Large Language Models
• Security Measures Validation
• Malicious Exploitation Prevention

Innovation

• AP-Test
• Safety Guardrails Detection
• Loss Function Design