Expert-Generated Privacy Q&A Dataset for Conversational AI and User Study Insights

2025-02-03

AI Summary

Current privacy notices suffer from legal imprecision, poor user comprehension, and low accessibility, hindering data transparency in conversational AI. To address this, we introduce the first interdisciplinary privacy Q&A dataset, collaboratively annotated by legal experts and conversational designers. Our approach establishes a novel cross-disciplinary annotation framework that jointly quantifies legal compliance, linguistic readability (Flesch-Kincaid grade level and lexical complexity), and user cognitive outcomes via controlled experiments. Compared to conventional privacy policies and commercial voice assistant responses (e.g., Alexa), our method achieves statistically significant improvements in user comprehension (p < 0.01) and trust. Legal accuracy reaches 100% expert consensus, while readability improves by 47% and 32% over baseline policies and Alexa responses, respectively. This work advances principled, human-centered design of privacy communication in conversational AI systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: AI for Accessibility
• Machine Learning: Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Conversational assistants process personal data and must comply with data protection regulations that require providers to be transparent with users about how their data is handled. Transparency, in a legal sense, demands preciseness, comprehensibility and accessibility, yet existing solutions fail to meet these requirements. To address this, we introduce a new human-expert-generated dataset for Privacy Question-Answering (Q&A), developed through an iterative process involving legal professionals and conversational designers. We evaluate this dataset through linguistic analysis and a user study, comparing it to privacy policy excerpts and state-of-the-art responses from Amazon Alexa. Our findings show that the proposed answers improve usability and clarity compared to existing solutions while achieving legal preciseness, thereby enhancing the accessibility of data processing information for Conversational AI and Natural Language Processing applications.

Problem

• Data Privacy
• Transparency
• AI Services

Innovation

• Privacy-related Q&A Database
• Transparency Enhancement
• Expert-designed Communication Framework