Assessing the Alignment of Popular CNNs to the Brain for Valence Appraisal

2025-09-23

AI Summary

This study investigates the alignment between mainstream convolutional neural networks (CNNs) and human brain activity associated with social cognition during image valence assessment—moving beyond low-level visual perception. Method: We propose Object2Brain, a framework integrating Grad-CAM interpretability analysis with object detection to quantify, at the filter level, the contribution of distinct object categories to CNN–brain functional alignment. We correlate CNN representations with human behavioral responses and fMRI data, focusing on higher-order social-cognitive brain regions. Results: We find that current CNNs generally fail to model high-level social-semantic processing: their representations exhibit significantly weaker alignment with activation in higher-order cortical areas (e.g., medial prefrontal cortex, temporoparietal junction) than with primary visual cortex. Moreover, architectural differences induce systematic variations in object-category sensitivity. This work is the first to reveal CNN limitations in social cognition at the fine-grained object–filter level, providing both theoretical foundations and an evaluation paradigm for developing socially intelligent vision models.

Technology Category

• Computer Vision: Object Detection & Categorization
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Convolutional Neural Networks (CNNs) are a popular type of computer model that have proven their worth in many computer vision tasks. Moreover, they form an interesting study object for the field of psychology, with shown correspondences between the workings of CNNs and the human brain. However, these correspondences have so far mostly been studied in the context of general visual perception. In contrast, this paper explores to what extent this correspondence also holds for a more complex brain process, namely social cognition. To this end, we assess the alignment between popular CNN architectures and both human behavioral and fMRI data for image valence appraisal through a correlation analysis. We show that for this task CNNs struggle to go beyond simple visual processing, and do not seem to reflect higher-order brain processing. Furthermore, we present Object2Brain, a novel framework that combines GradCAM and object detection at the CNN-filter level with the aforementioned correlation analysis to study the influence of different object classes on the CNN-to-human correlations. Despite similar correlation trends, different CNN architectures are shown to display different object class sensitivities.

Problem

• Evaluates CNN-brain alignment for social cognition tasks
• Assesses valence appraisal using behavioral and fMRI data
• Analyzes object class influence on neural correlations

Innovation

• Correlates CNNs with human behavioral data
• Uses GradCAM and object detection framework
• Analyzes object class influence on correlations