Breaking Bad Email Habits: Bounding the Impact of Simulated Phishing Campaigns

2026-03-04

AI Summary

Existing phishing simulation training suffers from endogeneity bias because it triggers only upon user clicks, conflating genuine habit formation with inherent individual differences and thereby inflating estimated intervention effects. This study proposes a generalizable causal inference framework that uniquely integrates marginal structural models (MSM) with correlated random effects (CRE), enabling simultaneous correction for endogenous training assignment and disentanglement of state dependence from individual heterogeneity using standard phishing logs. Empirical analysis of 17 rounds of university phishing campaigns (192,840 records) reveals that repeated clicking stems primarily from stable individual traits rather than behavioral inertia; affective framing and explicit reporting prompts significantly reduce click persistence, whereas annotated email cues slightly exacerbate it. Surprisingly, users who visited educational pages exhibited higher propensities for repeated clicking.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Content-based information diffusion
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Simulated phishing campaigns are widely deployed, yet the behavioral data they produce is endogenous: because training is triggered by clicking, the employees receiving intervention have already demonstrated susceptibility. This endogeneity, combined with the difficulty of separating genuine habit formation from stable individual differences, means standard analyses can mischaracterize program effectiveness. In this Research Note, we develop a generalizable analytic framework addressing both biases simultaneously. We utilize marginal structural models (MSMs) to correct for the endogenous, click-triggered assignment of training, while integrating correlated random effects (CRE) to disentangle true state dependence from stable employee heterogeneity. Applying the MSM+CRE estimator to logs from 17 campaigns delivered to university staff (192,840 observations) reveals that analyses ignoring stable differences overstate the causal persistence of clicking; most repeat clicking reflects who employees are, not the effect of recent failures. This persistence is context-dependent, amplifying when successive campaigns share persuasion cues. Teachable-moment features also matter: emotion framing and explicit reporting pitches can largely eliminate persistence, while annotated-email cues modestly exacerbate it. Finally, employees engaging with the education page exhibit greater persistence than those dismissing it, consistent with an emboldening mechanism. We contribute methodologically by integrating MSMs and CRE into a portable framework for analyzing standard simulation logs, and practically by identifying specific design levers so organizations can better sequence and evaluate their phishing programs.

Problem

• simulated phishing
• endogeneity
• habit formation
• individual heterogeneity
• causal persistence

Innovation

• marginal structural models
• correlated random effects
• simulated phishing
• behavioral persistence
• endogeneity