Traceable LLM Reasoning for Fake-Order Fraud Detection

2026-07-25

AI Summary

This work addresses the challenges of heavy reliance on handcrafted features and poor model interpretability in detecting fraudulent orders on large-scale O2O platforms. To this end, we propose DeepScrub, a novel framework that unifies heterogeneous risk signals into textual representations via a semantic alignment module and leverages domain-adaptive pretraining combined with a SURE (Suggest-and-Reflect) reasoning mechanism to enable high-accuracy, traceable fraud detection. By integrating expert feedback with model self-inspection, DeepScrub supports iterative refinement of its reasoning paths. Evaluated on real-world data, the method achieves a Macro-F1 score of 85.3%, outperforming the strongest baseline by 2.7 percentage points; notably, an 8B-parameter variant surpasses a 32B-parameter model. In online pilot deployment, it attains 91.8% precision and 88.5% recall, reducing manual initial review efforts by 94% and yielding annual cost savings of nearly one million RMB.

Abstract

Detecting fake-order fraud at scale remains a critical challenge for large online-to-offline (O2O) service platforms, as existing approaches often rely on expert-designed features, produce black-box decisions, and provide limited interpretability. To address these limitations, we propose DeepScrub, a reinforcement learning framework built upon large language models (LLMs) for fake-order fraud detection with traceable reasoning. DeepScrub introduces three innovations. First, a semantic unification module converts heterogeneous risk signals into textual descriptions that LLMs can understand. Second, continued pre-training on risk-control corpora injects domain knowledge, and task rewards jointly evaluate prediction correctness and reasoning quality. Third, the SUggest-REflect (SURE) mechanism incorporates expert feedback and model self-checking to iteratively refine reasoning paths. On a real-world fake-order fraud detection dataset, DeepScrub achieves a macro-F1 score of 85.3%, outperforming the best baseline by 2.7 percentage points. Our task-optimized 8B model further surpasses a 32B model, showing that domain adaptation can matter more than model scale in this setting. In a four-week live pilot, DeepScrub achieved 91.8% precision and 88.5% recall, improving over first-stage human reviewers by 16.6 and 38.8 percentage points. It reduced first-stage manual review workload by 94% and saved nearly one million RMB annually. These results show that DeepScrub improves fraud review accuracy, reduces first-stage review workload, and provides traceable evidence for production risk-review workflows.

Problem

• fake-order fraud detection
• interpretability
• black-box decisions
• online-to-offline platforms
• fraud detection at scale

Innovation

• traceable reasoning
• semantic unification
• reinforcement learning
• domain-adaptive pre-training
• SURE mechanism