Using Hierarchical Controlled Vocabularies to Understand CLIP Retrieval Failures in Historical Photo Collections

2026-07-22

AI Summary

This study investigates the instability of CLIP models in retrieving historical photographs from GLAM institution collections, a failure mode not previously explained through the lens of controlled vocabulary structure. For the first time, it correlates the AAT thesaurus’s root facets and hierarchical depth with CLIP’s retrieval performance. Through visual embedding clustering, image–text similarity evaluation, and fine-tuning experiments across three historical photo datasets, the work systematically analyzes CLIP’s visual consistency and image–text alignment. The findings reveal that these two properties are nearly uncorrelated and jointly define distinct failure types: root facets significantly influence visual consistency; image–text alignment for shallow-level terms is more amenable to improvement via fine-tuning; and terms exhibiting tight visual clusters yet deviant semantic labels yield the poorest retrieval performance.

Abstract

GLAM institutions (Galleries, Libraries, Archives, and Museums) organise image access using controlled vocabularies such as the Art and Architecture Thesaurus (AAT). For content-based image retrieval in these settings, vision-language models like CLIP are increasingly used, but their performance varies. This variation is known to relate to measures like concept abstraction and concept frequency. However, no prior work explains this variation in terms of the structural properties of vocabularies like the AAT that GLAM professionals already use. The AAT groups concepts into broad facets (Objects, Activities, Agents, etc.) and arranges terms hierarchically within them. In this paper, we ask whether two structural properties (root facet type and hierarchy depth) explain where CLIP retrieval succeeds and fails, and where fine-tuning helps. Across three historical photographic collections annotated with AAT terms, we examine visual coherence (whether a term's photographs cluster in CLIP's embedding space), text-image alignment (whether its label is near that cluster), and standard retrieval measures, which conflate the two. We find that visual coherence and text-image alignment are nearly uncorrelated across terms and jointly separate distinct failure modes. Terms whose photographs cluster tightly but whose label is distant from the cluster retrieve poorly in every collection, in two of three collections even worse than terms that fail on both metrics. We also show that while retrieval metrics do not correlate significantly with either structural property, root facet type does significantly separate categories with varying visual coherence. Finally, we find that fine-tuning improves retrieval overall, but its gains favour shallower terms in the hierarchy, where text-image alignment improves most, beyond what concept frequency explains.

Problem

• CLIP
• controlled vocabularies
• image retrieval
• hierarchical structure
• retrieval failure

Innovation

• controlled vocabularies
• CLIP
• visual coherence
• text-image alignment
• hierarchical structure