CityLoc: 6 DoF Localization of Text Descriptions in Large-Scale Scenes with Gaussian Representation

2025-01-15

AI Summary

This work addresses the challenging problem of fine-grained 6-degree-of-freedom (6DoF) camera pose estimation from natural language descriptions (e.g., “all traffic lights”) in urban scenes. We propose the first text-conditioned diffusion model that directly generates a *probability distribution* over poses—rather than a point estimate—enabling uncertainty-aware localization. Our method integrates CLIP-based cross-modal semantic alignment with differentiable 3D Gaussian splatting rendering feedback to establish a vision-language joint reasoning framework, and refines the pose distribution via reweighted sampling. Key contributions include: (i) the first application of diffusion models to text-driven 6DoF pose *distribution* modeling, overcoming limitations of retrieval-based and deterministic learning approaches; and (ii) state-of-the-art performance across five large-scale urban datasets, demonstrating superior accuracy, robustness, and capability in grounding abstract spatial concepts.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Localizing text descriptions in large-scale 3D scenes is inherently an ambiguous task. This nonetheless arises while describing general concepts, e.g. all traffic lights in a city. To facilitate reasoning based on such concepts, text localization in the form of distribution is required. In this paper, we generate the distribution of the camera poses conditioned upon the textual description. To facilitate such generation, we propose a diffusion-based architecture that conditionally diffuses the noisy 6DoF camera poses to their plausible locations. The conditional signals are derived from the text descriptions, using the pre-trained text encoders. The connection between text descriptions and pose distribution is established through pretrained Vision-Language-Model, i.e. CLIP. Furthermore, we demonstrate that the candidate poses for the distribution can be further refined by rendering potential poses using 3D Gaussian splatting, guiding incorrectly posed samples towards locations that better align with the textual description, through visual reasoning. We demonstrate the effectiveness of our method by comparing it with both standard retrieval methods and learning-based approaches. Our proposed method consistently outperforms these baselines across all five large-scale datasets. Our source code and dataset will be made publicly available.

Problem

• 6-DoF Localization
• Urban Environment
• Granular Object Detection

Innovation

• CityLoc
• Gaussian representations
• CLIP model