HyperFusion: Hierarchical Multimodal Ensemble Learning for Social Media Popularity Prediction

2025-07-01

AI Summary

Predicting social media post popularity is highly challenging due to strong coupling among multimodal factors—visual content, textual descriptions, temporal dynamics, and user behavioral signals. To address this, we propose a hierarchical multimodal fusion framework. First, we design a cross-modal similarity measurement mechanism to explicitly model inter-modal dependencies. Second, we introduce a pseudo-labeling-based two-stage training strategy to mitigate severe label scarcity. Third, we integrate CLIP for visual representation, Transformer-based models for text encoding, CatBoost/TabNet for structured metadata, and MLPs for user-specific features, enabling end-to-end joint modeling. Our approach achieves third place in the SMP Challenge 2025 Image Track and significantly outperforms all baselines on the SMP benchmark dataset. The implementation is publicly available.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Social media popularity prediction plays a crucial role in content optimization, marketing strategies, and user engagement enhancement across digital platforms. However, predicting post popularity remains challenging due to the complex interplay between visual, textual, temporal, and user behavioral factors. This paper presents HyperFusion, a hierarchical multimodal ensemble learning framework for social media popularity prediction. Our approach employs a three-tier fusion architecture that progressively integrates features across abstraction levels: visual representations from CLIP encoders, textual embeddings from transformer models, and temporal-spatial metadata with user characteristics. The framework implements a hierarchical ensemble strategy combining CatBoost, TabNet, and custom multi-layer perceptrons. To address limited labeled data, we propose a two-stage training methodology with pseudo-labeling and iterative refinement. We introduce novel cross-modal similarity measures and hierarchical clustering features that capture inter-modal dependencies. Experimental results demonstrate that HyperFusion achieves competitive performance on the SMP challenge dataset. Our team achieved third place in the SMP Challenge 2025 (Image Track). The source code is available at https://anonymous.4open.science/r/SMPDImage.

Problem

• Predicting social media post popularity with multimodal data
• Integrating visual, textual, temporal, and user behavioral factors
• Addressing limited labeled data through hierarchical ensemble learning

Innovation

• Hierarchical multimodal ensemble learning framework
• Three-tier fusion architecture integrating diverse features
• Two-stage training with pseudo-labeling and refinement