Multimodal Embeddings for 3D Similarity Search in Semantic Web-of-Things Digital-Twin Platforms

2026-08-03

AI Summary

This work addresses the challenge of integrating ontological logical constraints with multimodal similarity search in 3D scenes within Semantic Web of Things (SWoT) platforms. To bridge this gap, we extend the Thing'in platform with a multimodal embedding layer that encodes 3D point clouds, temporal attributes, and semantic labels into a unified latent vector space, jointly stored with a knowledge graph. Leveraging the Clock-G temporal graph database, we develop a hybrid query engine that enables 3D digital twin retrieval by combining graph-based filtering with vector similarity search. Experiments on the S3DIS dataset demonstrate that our approach effectively narrows the search space, and the pretrained embeddings not only support accurate similarity retrieval but also serve as effective general-purpose representations for downstream tasks. This study presents the first successful deep integration of multimodal embeddings with SWoT knowledge graphs.

Abstract

Semantic Web of Things (SWoT) platforms model physical infrastructure as knowledge graphs typed against domain ontologies, enabling expressive structural and logical queries. However, they lack native mechanisms to express similarity beyond strict ontological equivalence, which represents a critical gap for 3D digital twins in domains such as telecom infrastructure and industrial IoT, where queries must combine ontological constraints with multimodal similarity search over heterogeneous, temporally-evolving scene data. We propose a framework that extends SWoT platforms with a multimodal embedding layer: ontology-typed entities comprising 3D point clouds, temporal attributes, and semantic labels are encoded into latent vector representations stored alongside the knowledge graph, enabling hybrid ontology-vector queries that combine graph-based filtering with similarity search. Implemented on Orange Research's Thing'in platform with the Clock-G temporal graph database, a feasibility evaluation on S3DIS demonstrates that graph filtering effectively restricts the search pool under temporal and relational constraints, and that general-purpose pretrained encoders produce representations sufficient for similarity retrieval and as a preliminary encoding step for downstream predictive tasks.

Problem

• Semantic Web of Things
• 3D similarity search
• digital twin
• multimodal embeddings
• ontology

Innovation

• multimodal embeddings
• 3D similarity search
• Semantic Web of Things
• digital twin
• knowledge graph