PolicyGapper: Automated Detection of Inconsistencies Between Google Play Data Safety Sections and Privacy Policies Using LLMs

2026-04-17

AI Summary

This study addresses the frequent inconsistencies between data safety statements (DSS) on Google Play and apps’ privacy policies (PP), a problem exacerbated by the lack of automated verification methods that do not require access to application binaries. To bridge this gap, the authors propose PolicyGapper, a large language model–based four-stage pipeline—comprising crawling, preprocessing, analysis, and postprocessing—that enables, for the first time, fully automated detection of discrepancies in data practices between DSS and PP without binary access. Evaluated on 330 popular applications, PolicyGapper identified 2,689 instances of omitted disclosures. Manual validation on a subset yielded an average precision of 0.75, recall of 0.77, and F1 score of 0.76. The complete, reproducible toolchain is publicly released.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Privacy
• Application Domains: Misinformation & Fake News

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Mobile application developers are required to disclose how they collect, use, and share user data in compliance with privacy regulations. To support transparency, major app marketplaces have introduced standardized disclosure mechanisms. In 2022, Google mandated the Data Safety Section (DSS) on Google Play, requiring developers to summarize their data practices. However, compiling accurate DSS disclosures is challenging, as they must remain consistent with the corresponding privacy policy (PP), and no automated tool currently verifies this alignment. Prior studies indicate that nearly 80% of popular apps contain incomplete or misleading DSS declarations. We present PolicyGapper, an LLM-based methodology for automatically detecting discrepancies between DSS disclosures and privacy policies. PolicyGapper operates in four stages: scraping, pre-processing, analysis, and post-processing, without requiring access to application binaries. We evaluate PolicyGapper on a dataset of 330 top-ranked apps spanning all 33 Google Play categories, collected in Q3 2025. The approach identifies 2,689 omitted disclosures, including 2,040 related to data collection and 649 to data sharing. Manual validation on a stratified 10% subset, repeated across three independent runs, yields an average Precision of 0.75, Recall of 0.77, Accuracy of 0.69, and F1-score of 0.76. To support reproducibility, we release a complete replication package, including the dataset, prompts, source code, and results available at https://github.com/Mobile-IoT-Security-Lab/PolicyGapper and https://doi.org/10.5281/zenodo.19628493.

Problem

• Data Safety Section
• Privacy Policy
• Inconsistency Detection
• Mobile Applications
• Privacy Compliance

Innovation

• LLM-based inconsistency detection
• Data Safety Section
• Privacy Policy alignment
• Automated compliance verification
• Mobile app transparency