Learning visual representations for compositional analysis of artworks and photographs

2026-08-06

AI Summary

This work addresses the lack of effective and interpretable representations for visual composition by proposing a human-inspired compositional representation framework that integrates object-centric decomposition, perceptual grouping, and graph attention networks to model spatial relationships. The proposed method is systematically compared against large-scale pretrained models. Experimental results demonstrate that, under encoder-frozen settings, this interpretable approach achieves competitive performance in composition scoring prediction and image retrieval tasks. While data-driven large models attain superior performance given sufficient training data, they do so at the cost of interpretability and cross-domain generalization. This study thus reveals a critical trade-off among performance, generalization, and interpretability in computational composition understanding.

Abstract

Composition, the deliberate arrangement of visual elements, is central to how meaning, emotion, and aesthetic quality are conveyed in artwork, yet it remains among the least formalized dimensions of visual understanding. Prior work highlights a persistent gap in learning meaningful compositional representations, attributing it to semantic bias and suggesting that human-inspired approaches may be key. We compare two parallel paradigms for composition analysis: a human-inspired method grounded in perceptual grouping, and fine-tuned foundation models enabled by recent large-scale compositional datasets. The human-inspired approach uses object-centric models for region-level decomposition and a graph attention network to capture spatial relationships between elements. Both paradigms are evaluated on composition score/category prediction, compositional image retrieval, and visual saliency detection. With frozen encoders, the human-inspired method achieves competitive performance while remaining interpretable. When sufficient data enables fine-tuning, large self-supervised models outperform significantly, but at the cost of interpretability and cross-domain generalization.

Problem

• composition
• visual representation
• compositional analysis
• semantic bias
• aesthetic understanding

Innovation

• compositional representation
• perceptual grouping
• graph attention network
• foundation models
• interpretable AI