SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors

2025-05-30

AI Summary

This study addresses the emerging security threat of social engineering (SE) attacks synergistically driven by augmented reality (AR) and multimodal large language models (LLMs). Method: We construct the first synchronized multimodal SE attack dataset, comprising 180 annotated dialogues, AR-captured visual/audio cues, environmental context, and social profiles. We propose the first behavioral modeling framework for AR-LLM-coordinated attacks, incorporating quantifiable metrics for subjective trust and susceptibility. Our methodology integrates real-time AR sensing, synchronized multimodal annotation, speech-based sentiment analysis, facial micro-expression recognition, social profiling, and ethically compliant design. Contribution/Results: Empirical evaluation demonstrates high attack efficacy: 93.3% phishing link click-through rate, 85% telephone answer rate, and a 76.7% significant post-interaction trust increase—confirming exceptional stealth and manipulative power.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.

Problem

• Analyzing social engineering attacks via AR and LLMs
• Detecting multimodal adversarial manipulation in SE attacks
• Designing defensive frameworks against AR-driven SE threats

Innovation

• Multimodal dataset for AR-LLM social engineering
• Captures annotated conversations in adversarial scenarios
• Includes visual/audio cues and subjective metrics