Evaluating Apple Intelligence's Writing Tools for Privacy Against Large Language Model-Based Inference Attacks: Insights from Early Datasets

2025-06-04

AI Summary

This study empirically evaluates, for the first time, the effectiveness of Apple Intelligence’s on-device writing tools—specifically rewriting and tone adjustment—in safeguarding user privacy against large language model (LLM)-driven affective inference attacks. Method: Leveraging a curated dataset of sensitive emotional texts, we propose a novel dynamic rewriting paradigm targeting affective information neutralization; we systematically model LLM-based emotion recognition attacks and comparatively assess multiple rewriting strategies in mitigating emotional leakage risk. Results: Proper rewriting significantly reduces LLMs’ accuracy in inferring users’ original emotional states (average decrease of 32.7%), validating on-device text transformation as a lightweight, adaptive privacy-enhancing mechanism. This work establishes a reproducible evaluation framework, provides empirical evidence, and offers design guidelines for on-device AI privacy protection.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Emotional Intelligence

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The misuse of Large Language Models (LLMs) to infer emotions from text for malicious purposes, known as emotion inference attacks, poses a significant threat to user privacy. In this paper, we investigate the potential of Apple Intelligence's writing tools, integrated across iPhone, iPad, and MacBook, to mitigate these risks through text modifications such as rewriting and tone adjustment. By developing early novel datasets specifically for this purpose, we empirically assess how different text modifications influence LLM-based detection. This capability suggests strong potential for Apple Intelligence's writing tools as privacy-preserving mechanisms. Our findings lay the groundwork for future adaptive rewriting systems capable of dynamically neutralizing sensitive emotional content to enhance user privacy. To the best of our knowledge, this research provides the first empirical analysis of Apple Intelligence's text-modification tools within a privacy-preservation context with the broader goal of developing on-device, user-centric privacy-preserving mechanisms to protect against LLMs-based advanced inference attacks on deployed systems.

Problem

• Assessing Apple Intelligence's writing tools for privacy against LLM-based emotion inference attacks
• Evaluating text modifications' impact on LLM-based emotion detection in early datasets
• Developing on-device privacy mechanisms to neutralize sensitive emotional content dynamically

Innovation

• Apple Intelligence's text-modification tools for privacy
• Novel datasets to assess LLM-based detection
• Adaptive rewriting systems to neutralize emotional content