Modeling Scientific Experiment Scenes: Dataset and Model

2026-08-03

AI Summary

Existing scene graph generation benchmarks struggle to handle the challenges posed by scientific experimental settings, particularly concerning domain-specific instruments, task-oriented semantics, and dense, fine-grained physical relationships. To address this gap, this work introduces PhysScene, the first scene graph dataset tailored for physics experiments, along with a Cross-Modal Dual-Path Generator (CM-DPG). The proposed model integrates vision–language joint encoding, complementary geometric and visual relational reasoning, relation-aware pretraining, caption-based pseudo-supervision, and an adaptive loss weighting mechanism to effectively mitigate long-tailed relation distributions and the vision–language semantic gap. Experimental results demonstrate that CM-DPG achieves state-of-the-art performance on both PhysScene and VG150, significantly advancing structured understanding in scientific experimental scenarios.

Abstract

Scene Graph Generation (SGG) is fundamental to structured visual understanding, yet existing benchmarks focus mainly on daily life images and overlook scientific experiment scenes with specialized instruments, task-specific experimental semantics, and dense, fine-grained physical relations. These scenes are increasingly important for automated experimental analysis and smart education. To bridge this gap, we introduce PhysScene, the first SGG dataset for physical experiment scenes, providing densely annotated scene graphs and benchmarks under multiple supervision and protocol settings. PhysScene further exposes two key algorithmic challenges for SGG: pronounced long-tail relational predicate distributions and a substantial visual-textual semantic gap. To address these challenges, we propose the Cross-Modal Dual-Path Generator (CM-DPG), a model for robust open-vocabulary SGG. The model enhances object-level semantic representations through joint visual-textual encoding and improves relational reasoning using complementary visual and geometric cues. We also incorporate relation-aware pre-training, caption-derived pseudo-supervision, and adaptive weighting to support balanced learning across head and tail predicates. Extensive experiments on PhysScene and VG150 show that CM-DPG achieves competitive performance across multiple evaluation settings, with ablation studies validating the contribution of each component. The dataset and code are publicly available at https://github.com/ZMH-SDUST/CM-DPG.

Problem

• Scene Graph Generation
• scientific experiment scenes
• long-tail relations
• visual-textual semantic gap
• physical experiment

Innovation

• Scene Graph Generation
• Scientific Experiment Scenes
• Cross-Modal Learning
• Long-Tail Relations
• Open-Vocabulary SGG