Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking

2025-02-19

AI Summary

This work identifies a novel jailbreaking threat arising from structured-output interfaces in large language models (LLMs): adversaries with only API access can dynamically manipulate internal logits during generation to evade safety filters. To address this, the authors propose AttackPrefixTree (APT), the first black-box attack framework tailored for structured outputs. APT models both safety-rejecting prefixes and latent harmful-output prefixes, leveraging a prefix-tree structure, logit-space perturbation, and prompt engineering to enable online construction and optimization of attack patterns. Extensive experiments across multiple benchmark datasets demonstrate that APT substantially outperforms existing jailbreaking methods. The results expose a systemic vulnerability stemming from the coupling of safety mechanisms with structured-output generation, underscoring the urgent need to redesign LLM safety protocols to mitigate such interface-level exploits.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs.

Problem

• Enhancing jailbreak attacking in LLMs
• Exploiting structured output interfaces
• Bypassing safety measures with AttackPrefixTree

Innovation

• Exploits structured output interfaces
• Uses black-box attack framework
• Leverages safety refusal response prefixes