StrTransformer: Source-Wise Structured Transformers for Unsupervised Blind Source Recovery

2026-05-25

AI Summary

This work addresses the limitations in unsupervised blind source separation, particularly the absence of explicit structural priors and inadequate modeling of source trajectories. To this end, the authors propose StrTransformer, a novel framework featuring a source-level structured Transformer branch that directly optimizes the latent source matrix while preserving reconstruction consistency through an observation-space mixer. An ordered multi-scale controller enables each branch to specialize adaptively to distinct temporal scales, and differentiable structural constraints combined with ordered branch descriptors mitigate permutation symmetry. The architecture integrates locality-biased Transformers, multi-scale chunked tokens, random masked reconstruction, and structural regularization. Experimental results demonstrate that individual branches converge to distinct structural patterns across time scales, effectively recovering latent trajectories aligned with ground-truth sources.

Abstract

This paper proposes StrTransformer, a source-wise structured Transformer framework for blind source recovery and branch-wise latent modeling. Instead of using an encoder to infer latent variables, StrTransformer directly optimizes the latent source matrix together with an observation-space mixer and source-wise structural Transformer branches. The mixer enforces reconstruction consistency, while each Transformer branch imposes a differentiable structural constraint on one latent source trajectory. Specifically, each source is converted into multi-scale patch tokens, randomly masked, processed by a locality-biased Transformer, and evaluated through a masked patch reconstruction energy. This energy acts as an implicit source-wise structural prior. To encourage different latent branches to specialize into different temporal regimes, StrTransformer further introduces an ordered multi-scale controller that learns branch-specific patch-scale weights, ordered scale centers, and locality attention slopes. The resulting objective combines observation reconstruction, source-wise structural regularization, and modular auxiliary penalties for separation and scale specialization. We analyze the decoupling and coupling structure of the objective, the regularized exact-reconstruction fiber, and the reduction of permutation symmetry induced by ordered branch descriptors. A controlled case study shows that the learned branches converge to distinct temporal-scale structures and recover source-aligned latent trajectories under post-hoc evaluation.

Problem

• blind source recovery
• unsupervised learning
• structured Transformer
• latent modeling
• temporal-scale separation

Innovation

• Structured Transformer
• Blind Source Recovery
• Latent Source Modeling
• Masked Patch Reconstruction
• Ordered Multi-scale Control