When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills

2026-08-04

AI Summary

This work presents the first systematic evaluation of security risks—specifically privacy leakage and identity impersonation—associated with persona skills constructed from interaction histories. To this end, we introduce AntiSkillBench, an end-to-end benchmark comprising 50 behaviorally rich personas and 7,500 dialogue trajectories. Leveraging three skill distillation strategies and four defense mechanisms, our framework enables comprehensive, multi-level risk assessment spanning data, skill representations, and agent behaviors. Experimental results reveal that persona skills consistently exhibit cross-architecture and cross-method vulnerabilities to both privacy leakage and behavioral mimicry. Moreover, existing defenses demonstrate limited efficacy and poor generalization, underscoring the critical role of AntiSkillBench in advancing research on secure and controllable personalized agents.

Abstract

Persona skills distill personal interaction histories into portable and executable artifacts for downstream agents. While enabling flexible personalization, this process concentrates fragmented personal signals, amplifies their impact through reuse, and challenges defenses designed for individual records or retrieval-based memory. To systematically investigate the safety of the persona-skill pipeline, we introduce AntiSkillBench, an end-to-end benchmark for evaluating risks and defenses across the persona-skill pipeline. It comprises: (i) a dataset of 7,500 persona-grounded dialogue traces, constructed from 50 behaviorally rich profiles spanning diverse task scenarios; (ii) an evaluation suite that measures skill-level privacy leakage and agent-level attribute disclosure and behavioral impersonation across three skill-distillation strategies; and (iii) a defense evaluation covering four configurations across online and post-hoc interventions, including active risk suppression and passive provenance protection. Experiments across three frontier agents show that persona-skill risks persist across agent backbones and distillation protocols, extending from explicit attributes to communication styles and personality traits. Existing defenses exhibit limited and distillation-dependent effectiveness, failing to generalize across risk and distillation strategies. These results highlight AntiSkillBench as a challenging benchmark for developing privacy-preserving and authenticity-aware persona skills.

Problem

• privacy leakage
• impersonation risk
• persona skills
• defense evaluation
• agent safety

Innovation

• persona skills
• privacy leakage
• behavioral impersonation
• AntiSkillBench
• defense evaluation