Technology Category
Computer Vision: Large Vision ModelsKnowledge Representation and Reasoning: Geometric, Spatial, and Temporal ReasoningMachine Learning: Large Multimodal Models (LMMs)
GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design
🤖 AI Summary
High annotation cost and poor generalizability plague geometric feature labeling in engineering design images. Method: We propose a large-model-driven automated annotation framework featuring (1) GeoBiked—the first bicycle-structure-specific dataset (4,355 images); (2) Diffusion-Hyperfeatures, a novel representation technique enabling precise cross-image correspondence of geometric keypoints; (3) a dual-path GPT-4o input mechanism integrating image and category labels, enhanced by systematic prompt engineering and a structured annotation schema to ensure fidelity and consistency of technical descriptions; and (4) a multi-source image collaborative geometric localization strategy. Results: Experiments demonstrate significant improvements in keypoint detection accuracy on unseen samples, with high descriptive accuracy from GPT-4o outputs. The framework validates the feasibility and strong generalization capability of large language-vision models for engineering image understanding and fine-grained geometric annotation.
Technology Category
Application Category
📝 Abstract
We provide a dataset for enabling Deep Generative Models (DGMs) in engineering design and propose methods to automate data labeling by utilizing large-scale foundation models. GeoBiked is curated to contain 4 355 bicycle images, annotated with structural and technical features and is used to investigate two automated labeling techniques: The utilization of consolidated latent features (Hyperfeatures) from image-generation models to detect geometric correspondences (e.g. the position of the wheel center) in structural images and the generation of diverse text descriptions for structural images. GPT-4o, a vision-language-model (VLM), is instructed to analyze images and produce diverse descriptions aligned with the system-prompt. By representing technical images as Diffusion-Hyperfeatures, drawing geometric correspondences between them is possible. The detection accuracy of geometric points in unseen samples is improved by presenting multiple annotated source images. GPT-4o has sufficient capabilities to generate accurate descriptions of technical images. Grounding the generation only on images leads to diverse descriptions but causes hallucinations, while grounding it on categorical labels restricts the diversity. Using both as input balances creativity and accuracy. Successfully using Hyperfeatures for geometric correspondence suggests that this approach can be used for general point-detection and annotation tasks in technical images. Labeling such images with text descriptions using VLMs is possible, but dependent on the models detection capabilities, careful prompt-engineering and the selection of input information. Applying foundation models in engineering design is largely unexplored. We aim to bridge this gap with a dataset to explore training, finetuning and conditioning DGMs in this field and suggesting approaches to bootstrap foundation models to process technical images.
Problem
Research questions and friction points this paper is trying to address.
Enable Deep Generative Models for engineering design using geometric features.
Automate labeling of technical images with foundation models like GPT-4o.
Improve geometric correspondence detection in structural images using Hyperfeatures.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Automated labeling using large-scale foundation models
Diffusion-Hyperfeatures for geometric correspondence detection
GPT-4o for diverse technical image descriptions
🔎 Similar Papers
2024-02-27arXiv.orgCitations: 5
2024-03-01arXiv.orgCitations: 14
