MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation

2024-11-26

arXiv.org

AI Summary

Text-to-3D generation is hindered by the scarcity of high-quality paired text–3D data. To address this, we introduce the first ultra-large-scale, semantically aligned text–3D dataset—comprising over 40 million multi-level textual annotations across 8.9 million diverse 3D assets. We propose a novel multi-stage automatic annotation pipeline integrating multi-view vision-language models (VLMs) and large language models (LLMs), enhanced with human meta-information to suppress hallucination, and pioneer a dual-granularity annotation paradigm combining fine-grained descriptions with semantic tags. Furthermore, we design FX3D, a lightweight two-stage framework enabling end-to-end textured mesh generation in under 15 seconds. Our dataset achieves superior annotation quality, outperforming existing benchmarks by 72.41% (GPT-4 evaluation) and 73.40% (human evaluation) in pairwise preference tests. FX3D establishes a new trade-off between fidelity and inference speed, advancing practical text-to-3D synthesis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: 3D Computer Vision

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Generating high-fidelity 3D content from text prompts remains a significant challenge in computer vision due to the limited size, diversity, and annotation depth of the existing datasets. To address this, we introduce MARVEL-40M+, an extensive dataset with 40 million text annotations for over 8.9 million 3D assets aggregated from seven major 3D datasets. Our contribution is a novel multi-stage annotation pipeline that integrates open-source pretrained multi-view VLMs and LLMs to automatically produce multi-level descriptions, ranging from detailed (150-200 words) to concise semantic tags (10-20 words). This structure supports both fine-grained 3D reconstruction and rapid prototyping. Furthermore, we incorporate human metadata from source datasets into our annotation pipeline to add domain-specific information in our annotation and reduce VLM hallucinations. Additionally, we develop MARVEL-FX3D, a two-stage text-to-3D pipeline. We fine-tune Stable Diffusion with our annotations and use a pretrained image-to-3D network to generate 3D textured meshes within 15s. Extensive evaluations show that MARVEL-40M+ significantly outperforms existing datasets in annotation quality and linguistic diversity, achieving win rates of 72.41% by GPT-4 and 73.40% by human evaluators.

Problem

• Limited dataset size and diversity hinder high-fidelity text-to-3D generation.
• Automated multi-level annotation pipeline enhances 3D asset descriptions.
• Rapid 3D mesh generation with improved linguistic diversity and quality.

Innovation

• Multi-stage annotation pipeline with VLMs and LLMs
• Two-stage text-to-3D pipeline with Stable Diffusion
• Human metadata integration to reduce VLM hallucinations