The Pervasive Blind Spot: Benchmarking VLM Inference Risks on Everyday Personal Videos

2025-11-04

AI Summary

This study systematically investigates the privacy risks posed by vision-language models (VLMs) in inferring sensitive personal information—such as age, occupation, and health status—from everyday personal videos. Method: Leveraging a crowdsourced dataset of 508 authentic personal videos, we conduct a benchmark evaluation of mainstream VLMs using human baseline experiments and multi-dimensional prompting strategies. Contribution/Results: We find that VLMs significantly outperform humans in sensitive attribute inference (average +23.6% accuracy), with performance critically dependent on temporal behavioral modeling rather than static object recognition. Common objects frequently act as misleading confounders, inducing erroneous attributions. Moreover, model-generated explanations exhibit extremely low alignment with actual reasoning grounds (mean IoU < 0.18). These findings expose the “black-box reasoning” privacy vulnerability of VLMs in personal video contexts, providing empirical foundations and methodological insights for developing trustworthy VLMs and privacy-preserving mechanisms.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Privacy
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The proliferation of Vision-Language Models (VLMs) introduces profound privacy risks from personal videos. This paper addresses the critical yet unexplored inferential privacy threat, the risk of inferring sensitive personal attributes over the data. To address this gap, we crowdsourced a dataset of 508 everyday personal videos from 58 individuals. We then conducted a benchmark study evaluating VLM inference capabilities against human performance. Our findings reveal three critical insights: (1) VLMs possess superhuman inferential capabilities, significantly outperforming human evaluators, leveraging a shift from object recognition to behavioral inference from temporal streams. (2) Inferential risk is strongly correlated with factors such as video characteristics and prompting strategies. (3) VLM-driven explanation towards the inference is unreliable, as we revealed a disconnect between the model-generated explanations and evidential impact, identifying ubiquitous objects as misleading confounders.

Problem

• Evaluating VLM privacy risks from inferring sensitive personal attributes
• Benchmarking superhuman VLM inference capabilities against human performance
• Analyzing factors affecting inferential risks and unreliable model explanations

Innovation

• Benchmarked VLM inference risks using crowdsourced personal videos
• Revealed superhuman VLM capabilities in behavioral inference
• Identified unreliable explanations and misleading object confounders