🤖 AI Summary
Addressing the challenge of scarce target-domain data for complex real-world problems, this paper tackles the limitations of single-domain multimodal data fusion. Method: We propose a novel cross-domain multimodal knowledge fusion paradigm, formally defining the cross-domain multimodal fusion problem for the first time. Our approach introduces a four-layer framework—comprising domain, linkage, model, and data layers—that systematically addresses the core questions of *what to fuse*, *why fusion is feasible*, and *how to fuse*. Key techniques include fusion-aware knowledge graph alignment, cross-domain representation learning, multi-scale feature normalization, and mechanism-driven coupled/decoupled fusion models. Contribution/Results: The method enables end-to-end deployment and significantly enhances generalization and robustness in resource-constrained scenarios, demonstrating state-of-the-art performance on urban anomaly detection and industrial fault diagnosis tasks.
📝 Abstract
The proliferation of artificial intelligence has enabled a diversity of applications that bridge the gap between digital and physical worlds. As physical environments are too complex to model through a single information acquisition approach, it is crucial to fuse multimodal data generated by different sources, such as sensors, devices, systems, and people, to solve a problem in the real world. Unfortunately, it is neither applicable nor sustainable to deploy new resources to collect original data from scratch for every problem. Thus, when data is inadequate in the domain of problem, it is vital to fuse knowledge from multimodal data that is already available in other domains. We call this cross-domain knowledge fusion. Existing research focus on fusing multimodal data in a single domain, supposing the knowledge from different datasets is intrinsically aligned; however, this assumption may not hold in the scenarios of cross-domain knowledge fusion. In this paper, we formally define the cross-domain multimodal data fusion problem, discussing its unique challenges, differences and advantages beyond data fusion in a single domain. We propose a four-layer framework, consisting of Domains, Links, Models and Data layers, answering three key questions:"what to fuse","why can be fused", and"how to fuse". The Domains Layer selects relevant data from different domains for a given problem. The Links Layer reveals the philosophy of knowledge alignment beyond specific model structures. The Models Layer provides two knowledge fusion paradigms based on the fundamental mechanisms for processing data. The Data Layer turns data of different structures, resolutions, scales and distributions into a consistent representation that can be fed into an AI model. With this framework, we can design end-to-end solutions that fuse cross-domain multimodal data effectively for solving real-world problems.