EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detection

2026-07-24

AI Summary

This work addresses the challenge that existing harmful meme detection methods struggle to accurately interpret sarcasm or implicit semantics in multimodal content, primarily due to insufficient background knowledge, limited multi-perspective reasoning capabilities, and weak fine-grained image-text alignment. To overcome these limitations, the paper proposes an Enhanced Vision-Language Multi-Chain-of-Thought (EVL-MCoT) framework, which innovatively integrates multi-perspective chain-of-thought reasoning, prototype-guided fine-grained alignment, and context-guided decoding within a dual-stream vision-language architecture. This approach substantially enhances the model’s ability to comprehend complex meme semantics and improves judgment reliability. The method achieves state-of-the-art performance on the HatefulMemes and MultiOff benchmarks, and the implementation code has been publicly released.

Abstract

MEMEs are widely used on the internet and often carry strong elements of sarcasm or irony. Understanding their hidden meanings typically requires a joint interpretation of text and vision. Existing methods focus on the dual-stream vision-language model to extract the visual and text simultaneously, which lacks background information and prior knowledge about the comprehensive explanation of MEME. One feasible option is to adopt chain-of-thought (CoT). However, the simple CoT approach lacks multi-perspective thinking, which may compromise the reliability of the resulting answers. Moreover, it often relies on shallow feature fusion, lacking the fusion of local details and fine-grained visual-prompt text alignment. This limitation prevents a deeper understanding of the intricate connections between the visual and the text. Herein, an enhanced vision-language multi-CoT (EVL-MCoT) approach is proposed to address these limitations. By promoting multi-CoT, EVL-MCoT enhances consistency and reduces bias in the decision-making process. Additionally, we design a prototype-guided and context-guided decoding framework, which incorporates visual prototypes to guide the fusion process and enables the model to align textual and visual information more precisely. We achieve promising results on the HatefulMemes and MultiOff datasets. The source code has been publicly released and is available at https://github.com/BGWH123/EVL-MCoT.

Problem

• harmful meme detection
• vision-language understanding
• chain-of-thought
• feature fusion
• visual-text alignment

Innovation

• Multi-CoT
• vision-language alignment
• prototype-guided fusion
• harmful meme detection
• fine-grained multimodal reasoning