Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

2026-10-01

AI Summary

This study addresses the privacy risks inherent in personalized vision-language models, where adversaries can bind and recognize target identities using only a few images. To mitigate this threat, we propose Anti-Persona, a defense mechanism that optimizes visual perturbations to disrupt identity prototype alignment, thereby preventing unauthorized identity binding and recognition. Specifically, our method aggregates identity prototypes from shared visual features and incorporates spatial smoothing alongside low-frequency preservation techniques to achieve prompt-agnostic robust protection. Experimental results demonstrate that Anti-Persona attains a protection rate of up to 95.0% across representative models while maintaining high visual fidelity. Furthermore, the proposed approach exhibits strong black-box transferability across diverse visual encoders, highlighting its practical effectiveness against unauthorized personalization.

Abstract

Few-shot personalization enables large vision--language models (LVLMs) to learn user-specific visual concepts for applications such as personalized retrieval and subject-aware querying. However, it also creates a privacy risk: an adversary can bind a target identity from a few reference images and subsequently detect that identity in new images through natural-language queries. We introduce Anti-Persona, an image-level defense against unauthorized identity binding and recognition in personalized LVLMs. Our key insight is that identity personalization relies on visual features shared across multiple reference images. We aggregate these features into an identity prototype and optimize visually subtle perturbations that disrupt prototype alignment in the vision-encoder space. Spatial smoothing and low-frequency preservation further promote visual fidelity and practical resilience to image compression. The resulting protection does not depend on a specific prompt and supports both proactive anti-personalization and reactive image protection. Experiments on two representative personalized LVLMs demonstrate protection rates of up to $95.0\%$ while preserving visual fidelity. The method remains stable across prompt variations and evaluated identity-query tasks, and improves black-box transfer under encoder mismatch.

Problem

• privacy protection
• vision-language models
• unauthorized identity recognition
• few-shot personalization
• identity binding

Innovation

• Anti-Persona
• Vision-Language Models
• Identity Protection
• Adversarial Perturbation
• Few-shot Personalization