FMDLlama: Financial Misinformation Detection based on Large Language Models

2024-09-24

arXiv.org

AI Summary

To mitigate market risks arising from the rapid dissemination of financial rumors on social media, this paper introduces the first instruction-tuning framework for Financial Misinformation Detection (FMD). Methodologically, we build a specialized model based on Llama3.1 and construct FMDID—the first open, multi-task instruction dataset for FMD—covering both rumor classification and explanation generation. We further design FMD-B, a comprehensive evaluation benchmark, along with a rigorous annotation protocol. Our contributions include: (i) the first FMD-specific instruction-tuning paradigm; (ii) the first publicly available FMD instruction dataset; and (iii) the first integrated evaluation benchmark for FMD. Extensive experiments demonstrate that our model consistently outperforms leading open-source LLMs and commercial models from OpenAI on FMD-B. Both code and dataset are publicly released.

Technology Category

Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)Machine Learning: Large Multimodal Models (LMMs)Application Domains: Misinformation & Fake News

Application Category

Economics, Online Markets and Human Computation: LLM based quality controls for crowd workResponsible Web: Mitigating misinformation and disinformationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The emergence of social media has made the spread of misinformation easier. In the financial domain, the accuracy of information is crucial for various aspects of financial market, which has made financial misinformation detection (FMD) an urgent problem that needs to be addressed. Large language models (LLMs) have demonstrated outstanding performance in various fields. However, current studies mostly rely on traditional methods and have not explored the application of LLMs in the field of FMD. The main reason is the lack of FMD instruction tuning datasets and evaluation benchmarks. In this paper, we propose FMDLlama, the first open-sourced instruction-following LLMs for FMD task based on fine-tuning Llama3.1 with instruction data, the first multi-task FMD instruction dataset (FMDID) to support LLM instruction tuning, and a comprehensive FMD evaluation benchmark (FMD-B) with classification and explanation generation tasks to test the FMD ability of LLMs. We compare our models with a variety of LLMs on FMD-B, where our model outperforms other open-sourced LLMs as well as OpenAI's products. This project is available at https://github.com/lzw108/FMD.

Problem

• Large Language Models
• Financial Misinformation Detection
• Social Media

Innovation

• FMDLlama
• Financial Misinformation Detection
• Exceeds Existing Models