LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data

2025-08-16

AI Summary

This work addresses the lack of quantifiable standards for text privacy sensitivity assessment by investigating the feasibility of large language models (LLMs) as “privacy evaluators” and their alignment with human perception. Adopting the LLM-as-a-Judge paradigm, we conduct a multi-model, multi-task empirical study across 10 datasets, 13 LLMs, and 677 human participants, integrating quantitative evaluation with qualitative analysis. We first empirically demonstrate—via large-scale experimentation—that LLMs effectively model aggregate human privacy cognition (mean correlation = 0.72), significantly outperforming random baselines; moreover, despite low inter-annotator agreement among humans (Cohen’s κ = 0.38), LLMs robustly approximate population-level privacy sensitivity distributions. We further identify systematic discrepancies between human and LLM reasoning grounds and propose alignment-oriented optimization directions. This work establishes a novel paradigm for privacy-aware modeling and trustworthy NLP evaluation.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

Despite advances in the field of privacy-preserving Natural Language Processing (NLP), a significant challenge remains the accurate evaluation of privacy. As a potential solution, using LLMs as a privacy evaluator presents a promising approach $unicode{x2013}$ a strategy inspired by its success in other subfields of NLP. In particular, the so-called $ extit{LLM-as-a-Judge}$ paradigm has achieved impressive results on a variety of natural language evaluation tasks, demonstrating high agreement rates with human annotators. Recognizing that privacy is both subjective and difficult to define, we investigate whether LLM-as-a-Judge can also be leveraged to evaluate the privacy sensitivity of textual data. Furthermore, we measure how closely LLM evaluations align with human perceptions of privacy in text. Resulting from a study involving 10 datasets, 13 LLMs, and 677 human survey participants, we confirm that privacy is indeed a difficult concept to measure empirically, exhibited by generally low inter-human agreement rates. Nevertheless, we find that LLMs can accurately model a global human privacy perspective, and through an analysis of human and LLM reasoning patterns, we discuss the merits and limitations of LLM-as-a-Judge for privacy evaluation in textual data. Our findings pave the way for exploring the feasibility of LLMs as privacy evaluators, addressing a core challenge in solving pressing privacy issues with innovative technical solutions.

Problem

• Evaluating privacy sensitivity in textual data using LLMs
• Aligning LLM privacy evaluations with human perceptions
• Assessing feasibility of LLMs as privacy evaluators

Innovation

• LLM-as-a-Judge for privacy evaluation
• Measuring human-LLM privacy perception alignment
• Global human privacy perspective modeling