Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models

2026-03-23

AI Summary

Existing vision-language models struggle to effectively capture the hierarchical part-whole semantic relationships in images, particularly in complex multi-object compositional scenes. This work proposes UNCHA, a novel approach that introduces semantic representativeness-aware uncertainty modeling into hyperbolic vision-language alignment for the first time. By representing hierarchical structures through hyperbolic embeddings, UNCHA dynamically assigns uncertainty weights based on the semantic representativeness of local regions with respect to the whole image. The embedding space is optimized via a combination of contrastive learning, entailment loss, and entropy regularization. Extensive experiments demonstrate that UNCHA achieves state-of-the-art performance across zero-shot classification, image-text retrieval, and multi-label classification tasks, significantly enhancing the model’s ability to understand intricate multi-object scenes.

Technology Category

• Computer Vision: Language and Vision
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While Vision-Language Models (VLMs) have achieved remarkable performance, their Euclidean embeddings remain limited in capturing hierarchical relationships such as part-to-whole or parent-child structures, and often face challenges in multi-object compositional scenarios. Hyperbolic VLMs mitigate this issue by better preserving hierarchical structures and modeling part-whole relations (i.e., whole scene and its part images) through entailment. However, existing approaches do not model that each part has a different level of semantic representativeness to the whole. We propose UNcertainty-guided Compositional Hyperbolic Alignment (UNCHA) for enhancing hyperbolic VLMs. UNCHA models part-to-whole semantic representativeness with hyperbolic uncertainty, by assigning lower uncertainty to more representative parts and higher uncertainty to less representative ones for the whole scene. This representativeness is then incorporated into the contrastive objective with uncertainty-guided weights. Finally, the uncertainty is further calibrated with an entailment loss regularized by entropy-based term. With the proposed losses, UNCHA learns hyperbolic embeddings with more accurate part-whole ordering, capturing the underlying compositional structure in an image and improving its understanding of complex multi-object scenes. UNCHA achieves state-of-the-art performance on zero-shot classification, retrieval, and multi-label classification benchmarks. Our code and models are available at: https://github.com/jeeit17/UNCHA.git.

Problem

• Vision-Language Models
• Hyperbolic Embeddings
• Part-to-Whole Relationships
• Semantic Representativeness
• Compositional Understanding

Innovation

• Hyperbolic Vision-Language Models
• Part-to-Whole Representativeness
• Uncertainty-Guided Alignment
• Compositional Understanding
• Entailment Regularization