Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound

2026-09-23

AI Summary

该论文介绍了一个名为Passing的互动音视频装置，通过重构时空结构和使用AI生成的声音来创造一个不断变化的景观体验。

Abstract

This paper introduces Passing, an interactive audiovisual installation that generates an endless journey from a single continuous monorail-window recording by reconstructing it as a spatiotemporal volume. Rather than replaying the footage linearly, the work resamples its spatial and temporal structure along nonlinear trajectories, producing a continuously passing landscape whose depth, speed, and temporal order become unstable. A camera-based viewer-presence detection system estimates whether a viewer is present in the viewing zone and uses this presence state to influence transitions among rendered video sequences. The resulting video stream is fed into SpecMaskFoley, a real-time video-to-audio synthesis model that generates a synchronized soundscape for the reconfigured image. The model is not used to reconstruct an objectively correct soundtrack, but functions as a speculative listener, proposing a possible auditory interpretation of a world whose conventional spatial and temporal premises have been disrupted. Passing distributes creative agency across the artist, who defines the rules of spacetime reconstruction; the AI model, which interprets the emergent visual flow as sound; and the audience, whose embodied presence influences the audiovisual trajectory. Through this structure, the work investigates how authorship and listening may be negotiated among human intention, machine inference, and audience interpretation. Artwork page: https://ryufurusawa.com/passing

Problem

• authorship
• listening
• human intention
• machine inference
• audience interpretation

Innovation

• spatiotemporal volume
• nonlinear trajectories
• viewer-presence detection
• real-time video-to-audio synthesis