🤖 AI Summary
This study addresses the lack of empirical evidence on key operational metrics—such as latency, cost, fairness, and adversarial robustness—for deploying large language models (LLMs) in trust and safety workflows like fraud detection and content moderation. Through a systematic literature review of 49 operationally relevant studies, the authors propose the FORTE role framework and a minimal deployment evidence checklist encompassing dimensions like latency budgets and per-decision costs to structurally evaluate LLM roles and evidence completeness in real-world settings. The analysis reveals a structural imbalance in existing work: while content moderation exhibits relatively comprehensive operational evidence, fraud detection suffers from severe gaps. The paper concludes by outlining critical directions for empirical research necessary to support reliable LLM deployment in high-stakes applications.
📝 Abstract
LLMs are now proposed for fraud detection, scam investigation, content moderation, and other trust-and-safety workflows. Much of the public literature still evaluates them as models, with less attention to their behavior as components in operational pipelines. This creates a practical evidence question: what would justify placing an LLM inside a live workflow with latency, cost, escalation, human-review, and adversarial-risk constraints?
We address this question through a fraud-first survey of deployment evidence. We code 49 operationally relevant sources on LLM use in fraud detection, investigation support, content moderation, and cross-cutting robustness (18 fraud, 14 moderation, 17 cross-cutting), supplemented by 15 contextual references that establish the survey boundaries. These sources include systems, benchmarks, frameworks, and deployment-relevant surveys, not 49 production deployments.
The main finding is an evidence imbalance. Fraud supplies the largest task-specific portion of the coded corpus. The moderation papers, however, include more explicit public evidence on latency, cost, governance, and fairness. Among the 18 fraud and investigation sources, none report clean per-decision latency, per-decision dollar cost, or calibration evidence; most report offline task performance, retrieval gains, or case-study accuracy instead.
The survey contributes a role-and-evidence organizing frame, FORTE, for locating LLMs as classifiers, retrieval interfaces, explanation generators, reviewer assistants, agents, feature extractors, or escalation components. It also contributes a minimum deployment-evidence checklist covering latency budget, cost per decision, decision threshold, explanation integrity, and adversarial pressure. The resulting agenda identifies studies needed to support deployment claims for LLM-based fraud and trust-and-safety work.