SACRED: A Faithful Annotated Multimedia Multimodal Multilingual Dataset for Classifying Connectedness Types in Online Spirituality

2026-03-28

AI Summary

This study addresses the lack of high-quality, publicly available multimodal datasets for classifying connection types in online spiritual contexts. To bridge this gap, the authors collaborated with social scientists to construct SACRED—the first multilingual, multimedia, multimodal annotated dataset dedicated to this domain—and identified a novel connection type that extends communication theory. The annotation process employed an expert-in-the-loop framework, and the dataset was systematically evaluated using rule-based methods, fine-tuned models, and 13 state-of-the-art large language models. Experimental results demonstrate the effectiveness of both the dataset and the proposed approach: DeepSeek-V3 achieved 79.19% accuracy on the Quora text test set, while GPT-4o-mini attained an F1 score of 63.99% on visual tasks.

Abstract

In religion and theology studies, spirituality has garnered significant research attention for the reason that it not only transcends culture but offers unique experience to each individual. However, social scientists often rely on limited datasets, which are basically unavailable online. In this study, we collaborated with social scientists to develop a high-quality multimedia multi-modal datasets, \textbf{SACRED}, in which the faithfulness of classification is guaranteed. Using \textbf{SACRED}, we evaluated the performance of 13 popular LLMs as well as traditional rule-based and fine-tuned approaches. The result suggests DeepSeek-V3 model performs well in classifying such abstract concepts (i.e., 79.19\% accuracy in the Quora test set), and the GPT-4o-mini model surpassed the other models in the vision tasks (63.99\% F1 score). Purportedly, this is the first annotated multi-modal dataset from online spirituality communication. Our study also found a new type of connectedness which is valuable for communication science studies.

Problem

• spirituality
• connectedness classification
• multimodal dataset
• online communication
• annotated data

Innovation

• multimodal dataset
• online spirituality
• connectedness classification
• faithful annotation
• LLM evaluation