MARMOT: Masked Autoencoder for Modeling Transient Imaging

2025-06-10

AI Summary

Existing non-line-of-sight (NLOS) transient imaging methods heavily rely on handcrafted priors, limiting generalizability. This work introduces MARMOT, the first self-supervised masked autoencoder specifically designed for transient imaging. To enable flexible sampling of photon time-of-arrival sequences, we propose a Scan Pattern Mask (SPM) that emulates arbitrary temporal subsampling. Leveraging the large-scale TransVerse synthetic dataset—comprising 500K diverse 3D scenes—we conduct extensive pretraining. MARMOT adopts a Transformer-based encoder-decoder architecture, eliminating dependence on manual priors during reconstruction. It supports zero-shot feature transfer and lightweight decoder fine-tuning. Evaluated across multiple NLOS reconstruction benchmarks, MARMOT consistently outperforms state-of-the-art methods, achieving significant improvements in quantitative metrics (e.g., PSNR, SSIM). Qualitative results further demonstrate its strong generalization to complex, cluttered hidden scenes and high-fidelity reconstruction capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Pretrained models have demonstrated impressive success in many modalities such as language and vision. Recent works facilitate the pretraining paradigm in imaging research. Transients are a novel modality, which are captured for an object as photon counts versus arrival times using a precisely time-resolved sensor. In particular for non-line-of-sight (NLOS) scenarios, transients of hidden objects are measured beyond the sensor's direct line of sight. Using NLOS transients, the majority of previous works optimize volume density or surfaces to reconstruct the hidden objects and do not transfer priors learned from datasets. In this work, we present a masked autoencoder for modeling transient imaging, or MARMOT, to facilitate NLOS applications. Our MARMOT is a self-supervised model pretrianed on massive and diverse NLOS transient datasets. Using a Transformer-based encoder-decoder, MARMOT learns features from partially masked transients via a scanning pattern mask (SPM), where the unmasked subset is functionally equivalent to arbitrary sampling, and predicts full measurements. Pretrained on TransVerse-a synthesized transient dataset of 500K 3D models-MARMOT adapts to downstream imaging tasks using direct feature transfer or decoder finetuning. Comprehensive experiments are carried out in comparisons with state-of-the-art methods. Quantitative and qualitative results demonstrate the efficiency of our MARMOT.

Problem

• Modeling transient imaging for NLOS scenarios
• Learning features from masked transients via Transformer
• Pretraining on diverse datasets for downstream tasks

Innovation

• Masked autoencoder for transient imaging modeling
• Self-supervised pretraining on diverse NLOS datasets
• Transformer-based encoder-decoder with scanning pattern mask