Cog2Gen3D: Sculpturing 3D Semantic-Geometric Cognition for 3D Generation

2026-03-06

AI Summary

Existing 3D generation methods struggle to simultaneously ensure semantic plausibility and geometric consistency due to their inadequate modeling of absolute geometric constraints. To address this limitation, this work proposes Cog2Gen3D, which uniquely integrates absolute geometry and semantic information into a unified 3D cognitive graph. The method introduces a semantic-geometry dual-stream architecture that leverages multimodal feature embedding, cross-attention fusion, and a cognition-guided latent diffusion mechanism to generate 3D Gaussians with both structurally coherent geometry and semantically faithful content. Evaluated on the Marble World Labs validation set, Cog2Gen3D significantly outperforms current state-of-the-art approaches in both semantic fidelity and geometric reasonableness.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Generative models have achieved success in producing semantically plausible 2D images, but it remains challenging in 3D generation due to the absence of spatial geometry constraints. Typically, existing methods utilize geometric features as conditions to enhance spatial awareness. However, these methods can only model relative relationships and are prone to scale inconsistency of absolute geometry. Thus, we argue that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world. In this work, we propose Cog2Gen3D, a 3D cognition-guided diffusion framework for 3D generation. Our model is guided by three key designs: 1) Cognitive Feature Embeddings. We encode different modalities into semantic and geometric representations and further extract logical representations. 2) 3D Latent Cognition Graph. We structure different representations into dual-stream semantic-geometric graphs and fuse them via common-based cross-attention to obtain a 3D cognition graph. 3) Cognition-Guided Latent Diffusion. We leverage the fused 3D cognition graph as the condition to guide the latent diffusion process for 3D Gaussian generation. Under this unified framework, the 3D cognition graph ensures the physical plausibility and structural rationality of 3D generation. Moreover, we construct a validation subset based on the Marble World Labs. Extensive experiments demonstrate that our Cog2Gen3D significantly outperforms existing methods in both semantic fidelity and geometric plausibility.

Problem

• 3D generation
• semantic-geometric cognition
• absolute geometry
• spatial constraints
• physical plausibility

Innovation

• 3D generation
• semantic-geometric cognition
• latent diffusion
• cognitive graph
• absolute geometry