HMID-Net: An Exploration of Masked Image Modeling and Knowledge Distillation in Hyperbolic Space

2025-07-13

AI Summary

This work addresses two key limitations in visual-semantic hierarchical modeling: low modeling efficiency and restricted representational capacity of Euclidean embeddings. To this end, we propose HMID-Net—the first unified framework that jointly performs masked image modeling (MIM) and knowledge distillation (KD) in hyperbolic space. Methodologically, we design hyperbolic-adapted masked reconstruction objectives and a novel hyperbolic knowledge distillation loss to explicitly capture hierarchical semantic relationships. Theoretically, we leverage the tree-like metric properties of hyperbolic geometry to enhance hierarchical awareness. Extensive experiments on image classification and cross-modal retrieval demonstrate that HMID-Net significantly outperforms strong baselines—including MERU and CLIP—achieving superior hierarchical representation learning, improved training efficiency, and enhanced downstream generalization.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Visual and semantic concepts are often structured in a hierarchical manner. For instance, textual concept `cat' entails all images of cats. A recent study, MERU, successfully adapts multimodal learning techniques from Euclidean space to hyperbolic space, effectively capturing the visual-semantic hierarchy. However, a critical question remains: how can we more efficiently train a model to capture and leverage this hierarchy? In this paper, we propose the extit{Hyperbolic Masked Image and Distillation Network} (HMID-Net), a novel and efficient method that integrates Masked Image Modeling (MIM) and knowledge distillation techniques within hyperbolic space. To the best of our knowledge, this is the first approach to leverage MIM and knowledge distillation in hyperbolic space to train highly efficient models. In addition, we introduce a distillation loss function specifically designed to facilitate effective knowledge transfer in hyperbolic space. Our experiments demonstrate that MIM and knowledge distillation techniques in hyperbolic space can achieve the same remarkable success as in Euclidean space. Extensive evaluations show that our method excels across a wide range of downstream tasks, significantly outperforming existing models like MERU and CLIP in both image classification and retrieval.

Problem

• Efficiently train models to capture visual-semantic hierarchy in hyperbolic space
• Integrate Masked Image Modeling and knowledge distillation in hyperbolic space
• Improve performance in image classification and retrieval tasks

Innovation

• HMID-Net integrates MIM in hyperbolic space
• Introduces hyperbolic-specific distillation loss function
• Combines knowledge distillation with hyperbolic learning