Geometric Configurations of Perturbed Jailbreak Prompts

2026-07-22

AI Summary

This study addresses the security threat posed by string-level perturbation-based jailbreaking attacks on small-scale large language models by systematically investigating the relationship between their internal representations and output behaviors. Focusing on Qwen and Llama family models, the authors analyze the embedding space of the final token in the last layer and the geometric structure of the probability distributions over the top-50 next tokens. The work reveals, for the first time, that in refusal-dominant responses, neither representation space exhibits a clear behavioral hyperplane; the embedding space primarily separates inputs by spelling and formatting, while the probability space, though approximately one-dimensional, resists meaningful clustering. Notably, only a few tokens—such as “Sure” and “,”—show significant association with compliant responses in specific models. These findings offer novel insights into jailbreaking mechanisms and pathways toward enhancing model robustness.

Abstract

Perturbation techniques that turn unsuccessful jailbreak prompts into successful ones are continuously evolving, constituting a major security threat to LLM safety. In this paper, we investigate the internal representations of such string-level perturbed jailbreak inputs in the small weight models of the Qwen-2.5-1.5B/-3B/-7B-Instruct and Llama-3.2-1B/-3B/-3.1-8B-Instruct families. We select two representation spaces: the last-layer-last-token embedding space and the top-50 next-token probability space. The former space separates prompts based on their spelling and format, while the latter space is effectively one-dimensional but appears more complex to cluster. Within our refusal-dominated answer set we find no behavioral hyperplane in either space. Only the next token "Sure" in the 1.5B Qwen model, and both tokens "," and "ĊĊ" in the 1$ Llama model, display a significant association with a compliant-labeled answer.

Problem

• jailbreak prompts
• perturbation
• internal representations
• LLM safety
• geometric configurations

Innovation

• jailbreak prompts
• geometric representation
• embedding space
• next-token probability
• LLM safety