Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes

2026-07-16

AI Summary

This work addresses the challenges of multimodal understanding and interpretability arising from the coexistence of humor and harmful content in internet memes. It proposes a theory-driven, structured parsing framework that integrates twelve perspectives grounded in humor and hate speech theories. By incorporating a character-aware soft-gated attention mechanism and a prototype-based classifier, the model achieves accurate prediction while generating contextually relevant, transparent explanations. This approach represents the first effort to unify multi-perspective, theory-guided reasoning with explainability in meme analysis. Evaluated on the PrideMM and Memotion datasets, it attains 80.3% accuracy in humor detection and 75.9% in hate speech detection—significantly outperforming existing methods. Moreover, the generated explanations demonstrate high coherence and credibility, as validated by both human evaluators and GPT-4 assessments.

Abstract

Internet memes intertwine visual cues, textual content, and cultural context, making them particularly challenging to interpret in scenarios where humor, sarcasm, and harmful intent coexist. These complexities highlight the need for explainable meme understanding systems that can provide reliable and structured reasoning to support both accurate classification and human interpretability. However, existing multimodal classifiers either overlook these interdependencies or provide only limited interpretability. In this paper, we introduce MAR-12, a novel framework that leverages Vision Language Models (VLMs) for meme detection and understanding in settings where humorous and hateful elements may coexist. The framework first interprets each meme through twelve structured perspectives derived from humor and hate theories. It then applies a role-aware soft-gated attention mechanism to learn how much each perspective should contribute, followed by a prototype-based classifier for the final prediction. Finally, explanations are synthesized using both perspective-specific reasoning and learned attention weights, ensuring transparent and context-grounded justifications. We evaluate MAR-12 on the PrideMM and Memotion datasets, where it achieves up to 80.3% accuracy for humor detection and 75.9% accuracy for hate detection, outperforming state-of-the-art approaches. Furthermore, both human and GPT-4-based evaluations confirm that MAR-12 produces coherent and persuasive explanations, particularly for memes in which humorous and harmful cues co-occur.

Problem

• harmful humor
• meme understanding
• multimodal classification
• explainability
• humor and hate detection

Innovation

• multi-angle reasoning
• vision language models
• soft-gated attention
• prototype-based classification
• explainable AI