Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

2026-05-30

AI Summary

This work addresses the limitations of vision foundation models, which suffer from the quadratic complexity of self-attention mechanisms, leading to inefficiency in processing high-resolution images and prohibitively expensive pretraining. The authors present the first extension of two-dimensional Global Spatial Propagation Networks (GSPN) to foundation-model scale, introducing an efficient propagation module that integrates shared-memory tiling with channel compression. To circumvent costly from-scratch training, they devise a two-stage cross-architecture knowledge distillation strategy. The resulting vision encoder reduces model parameters by 15% while achieving a 2.1% absolute improvement on ADE20K semantic segmentation. It also substantially lowers data requirements for high-resolution transfer and delivers a 4× end-to-end speedup at 2K resolution per inference pass.

Abstract

Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear attention and state-space models reduce this cost, but often serialize images into 1D token streams and weaken the 2D spatial structure important for vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional embeddings, but have seen little use as foundation-scale encoders. We present C-GSPN, a foundation-scale vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak memory bandwidth and running up to 40--52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator distillation recipe that trains the new architecture from an attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference.

Problem

• vision foundation models
• self-attention
• quadratic complexity
• 2D spatial structure
• subquadratic alternatives

Innovation

• C-GSPN
• 2D spatial propagation
• subquadratic vision encoder
• CUDA kernel optimization
• cross-operator distillation