SeqLLM: Augmenting LLMs with Behavioral-Sequence Modeling for High-Stakes Decisions at WeChat Pay

2026-08-03

AI Summary

This work addresses the high false-positive rates in merchant risk control on platforms such as WeChat Pay, where conventional approaches struggle to jointly model massive-scale merchants’ textual descriptions and long-horizon behavioral sequences. To overcome this limitation, we propose SeqLLM, a novel framework that encodes discrete behavioral events as native tokens and effectively integrates the language understanding and sequential modeling capabilities of large language models. This integration is achieved through a lightweight semantic projector, prefix-guided supervised fine-tuning, and a two-stage alignment training strategy, which collectively mitigate catastrophic forgetting. Deployed in production, SeqLLM improves merchant screening accuracy from 92.0% to 97.5% and increases Precision@Top-0.01% for fraud detection by 26.8 percentage points. The method also achieves significant gains over strong baselines like User-LLM on benchmark datasets including MovieLens and Amazon.

Abstract

Merchant risk control at large payment platforms screens tens of millions of merchants daily, where false positives harm legitimate merchants and false negatives leave harmful activity undetected. The hardest cases require jointly understanding a merchant's textual profile and long behavioral sequence. Large language models (LLMs) excel at text but cannot natively model such sequences, while adapting them often causes catastrophic forgetting. We present SeqLLM, a framework that adds behavioral-sequence modeling to a pretrained LLM while preserving its language ability. SeqLLM combines three components: a compact discrete vocabulary that represents behavioral events as native tokens; a lightweight projector, trained with a two-stage alignment curriculum, that grounds these tokens in the LLM's semantic space; and prefix-guided capability injection, which acquires sequence-modeling ability through task-prefixed supervised fine-tuning rather than continual pre-training. SeqLLM is deployed at WeChat Pay, screening millions of merchants daily. Against the production DeepSeek-based LLM baseline, it raises screening precision from 92.0% to 97.5%. Its pretrained behavior-token embeddings also improve Precision@Top-0.01% by 26.8 percentage points in a production fraud detector serving billion-scale transaction traffic. Beyond payments, SeqLLM achieves state-of-the-art results on public recommendation benchmarks. On MovieLens and Amazon, it surpasses the strong User-LLM baseline by up to 32% relative Recall@5 while retaining markedly stronger language ability. On RecIF, it improves Pass@32 by 14.2% over the full OneRec-8B pipeline using only one-fifth of its GPU-days.

Problem

• merchant risk control
• behavioral-sequence modeling
• large language models
• false positives
• false negatives

Innovation

• behavioral-sequence modeling
• large language models
• discrete tokenization
• prefix-guided fine-tuning
• catastrophic forgetting mitigation