The Lift Spectrum: How Measurement-to-Space Adaptivity Shapes Robustness in Image-Free Single-Pixel Sensing

2026-07-24

AI Summary

This work addresses the overlooked yet critical role of the lifting operation—mapping 1D measurement sequences to 2D representations—in image-agnostic single-pixel sensing. Challenging the conventional practice of treating lifting as a simple reshape, the study systematically formalizes the lift design space and introduces a spectrum of lifting strategies based on input adaptivity. To enable end-to-end optimization, the authors propose the STSF network coupled with Task-Prioritized Loss Scheduling (TPLS). The architecture integrates a recurrent encoder, a cross-attention-based lifting mechanism, and a U-Net++ decoder. Evaluated at an ultra-low sampling rate of 3.13%, the method surpasses existing baselines by 3.2–9.9 mIoU across three datasets and maintains robustness even at 0.39% sampling. Notably, it deploys directly on real hardware without fine-tuning, achieving inference in approximately 14 ms per frame.

Abstract

Single-pixel sensing encodes a scene as a short sequence of coded measurements, and image-free methods infer the task directly from that sequence. Removing reconstruction does not remove the difficulty: it relocates it to the lift, the map from 1D measurements to a 2D representation, which prior work treats as a trivial reshape. We recast the lift as the central design axis of image-free sensing and order methods by how strongly it adapts to its input: a fixed-physics inverse (reconstruct-then-segment), a learned static projection, or a content-adaptive retrieval; position on this lift spectrum predicts behavior as acquisition degrades. The spatiotemporal soft-fusion (STSF) network pairs a probe-selected recurrent encoder with a cross-attention lift chosen by a parameter-matched ablation, ahead of its U-Net++ decoder, and trains under task-prioritized loss scheduling (TPLS), a scheduled reconstruction prior. In simulation, STSF+TPLS surpasses the prior image-free baseline on three datasets at 3.13% sampling (+3.2 to +9.9 pp foreground mIoU) and plateaus down to 0.39%. The strongest clean-trained reconstruct-then-segment baseline wins the noiseless limit, but under calibrated measurement noise image-free inference overtakes it, for a measured reason: the reconstruction pipeline amplifies the identical measurement noise before its segmenter reads it. Each region fails in its own signature: collapse, imprinting, or coarsening. STSF+TPLS transfers without fine-tuning to a real single-pixel bench as a proof of concept, at about 14 ms per mask. Charting the lift turns a scattered design space into a map of which lift to deploy at each operating point. Code and pretrained weights: https://github.com/Hanyuyuan6/STSF-TPLS

Problem

• single-pixel sensing
• image-free inference
• lift
• robustness
• measurement-to-space mapping

Innovation

• Lift Spectrum
• image-free sensing
• content-adaptive retrieval
• task-prioritized loss scheduling
• single-pixel imaging