MK-Pose: Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning

2025-07-09

AI Summary

This work addresses category-level 6D object pose estimation without explicit shape priors—i.e., given only the object category and no instance-specific geometry. We propose a multimodal keypoint learning framework that jointly leverages RGB images, point clouds, and category-level textual descriptions. Methodologically, we introduce an attention-driven query generation mechanism, a soft heatmap matching strategy, and a graph-based relational modeling module to enable cross-modal feature alignment and structured reasoning; additionally, self-supervised keypoint detection and graph-enhanced feature fusion support end-to-end joint optimization. Evaluated on CAMERA25, REAL275, and HouseCat6D benchmarks, our approach significantly outperforms prior art in both IoU and average precision, while demonstrating strong robustness to occlusion and superior cross-instance generalization. To our knowledge, this is the first method achieving high-accuracy category-level pose estimation without requiring explicit geometric priors.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Category-level object pose estimation, which predicts the pose of objects within a known category without prior knowledge of individual instances, is essential in applications like warehouse automation and manufacturing. Existing methods relying on RGB images or point cloud data often struggle with object occlusion and generalization across different instances and categories. This paper proposes a multimodal-based keypoint learning framework (MK-Pose) that integrates RGB images, point clouds, and category-level textual descriptions. The model uses a self-supervised keypoint detection module enhanced with attention-based query generation, soft heatmap matching and graph-based relational modeling. Additionally, a graph-enhanced feature fusion module is designed to integrate local geometric information and global context. MK-Pose is evaluated on CAMERA25 and REAL275 dataset, and is further tested for cross-dataset capability on HouseCat6D dataset. The results demonstrate that MK-Pose outperforms existing state-of-the-art methods in both IoU and average precision without shape priors. Codes will be released at href{https://github.com/yangyifanYYF/MK-Pose}{https://github.com/yangyifanYYF/MK-Pose}.

Problem

• Estimating object poses without instance-specific prior knowledge
• Overcoming occlusion and generalization challenges in pose estimation
• Integrating multimodal data for improved category-level pose accuracy

Innovation

• Multimodal RGB, point cloud, and text integration
• Self-supervised keypoint detection with attention
• Graph-enhanced local and global feature fusion