TransiT: Transient Transformer for Non-line-of-sight Videography

2025-03-14

AI Summary

To address the challenge of balancing image quality and real-time performance in Non-Line-of-Sight (NLOS) imaging at high frame rates, this paper proposes TransiT, a novel transient Transformer architecture that— for the first time—directly compresses the temporal dimension of femtosecond-scale transient data. TransiT incorporates a spatio-temporal feature fusion mechanism and a sparse transient modeling strategy, while leveraging transfer learning to bridge the domain gap between synthetic and real-world measurements. Under sparse sampling (16×16 grid) and rapid scanning (0.4 ms per point), the system achieves real-time reconstruction of 64×64 NLOS video at 10 fps. Quantitative evaluation shows substantial improvements in signal-to-noise ratio and structural fidelity over state-of-the-art methods. This work delivers the first end-to-end NLOS video reconstruction framework that simultaneously achieves high frame rate and high fidelity, enabling practical deployment in critical applications such as autonomous navigation and post-disaster search-and-rescue.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

High quality and high speed videography using Non-Line-of-Sight (NLOS) imaging benefit autonomous navigation, collision prevention, and post-disaster search and rescue tasks. Current solutions have to balance between the frame rate and image quality. High frame rates, for example, can be achieved by reducing either per-point scanning time or scanning density, but at the cost of lowering the information density at individual frames. Fast scanning process further reduces the signal-to-noise ratio and different scanning systems exhibit different distortion characteristics. In this work, we design and employ a new Transient Transformer architecture called TransiT to achieve real-time NLOS recovery under fast scans. TransiT directly compresses the temporal dimension of input transients to extract features, reducing computation costs and meeting high frame rate requirements. It further adopts a feature fusion mechanism as well as employs a spatial-temporal Transformer to help capture features of NLOS transient videos. Moreover, TransiT applies transfer learning to bridge the gap between synthetic and real-measured data. In real experiments, TransiT manages to reconstruct from sparse transients of $16 imes 16$ measured at an exposure time of 0.4 ms per point to NLOS videos at a $64 imes 64$ resolution at 10 frames per second. We will make our code and dataset available to the community.

Problem

• Achieves real-time NLOS imaging with high frame rates.
• Balances frame rate and image quality in NLOS videography.
• Uses Transient Transformer to reduce computation and improve SNR.

Innovation

• Transient Transformer compresses temporal input dimensions
• Feature fusion and spatial-temporal Transformer capture NLOS features
• Transfer learning bridges synthetic and real data gaps