SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation

2026-05-31

AI Summary

This work addresses the representational gap arising from architectural disparities when distilling large vision Transformers into lightweight convolutional networks, a challenge that hinders effective transfer of global semantic relationships and spatial structures. To bridge this gap, the authors propose SWARD, a novel framework that jointly optimizes spatial dependency and feature separability in heterogeneous knowledge distillation. Specifically, SWARD employs Multi-scale Window Attention Distillation (MWAD) with random shifts to align spatial relationships between teacher and student networks while mitigating window boundary artifacts. Additionally, it introduces Prototype Discriminative Regularization (PDR) to refine the student’s feature distribution. Extensive experiments demonstrate that SWARD achieves state-of-the-art performance on urban scene parsing and medical image segmentation benchmarks, significantly outperforming existing distillation approaches.

Abstract

Large-scale vision foundation models have driven substantial gains on dense prediction tasks such as semantic segmentation, but their size makes deployment impractical in resource-constrained settings, motivating knowledge distillation as a means of transferring their capabilities to lightweight student networks. However, modern foundation teachers are predominantly transformer-based that encode global context, whereas efficient students are typically convolutional networks with locally biased receptive fields. Existing distillation methods largely assume architectural homogeneity and rely on direct feature mimicry, which fails to bridge this representational gap and neglects the structured spatial dependencies and discriminative organization required for accurate semantic segmentation. In this paper, we propose SWARD, a knowledge distillation framework that addresses this gap through two complementary mechanisms. First, we introduce a Multi-Scale Windowed Attention Distillation (MWAD) module that aligns teacher-student attention-based relations within stochastically shifted window partitions whose offsets are randomly resampled at every training iteration. This removes window boundary bias, and, combined with the multi-scale design, captures both short- and long-range spatial dependencies. Second, we introduce Prototype Discriminative Regularization (PDR), a loss that helps shape the student's feature distribution by enforcing inter-class separation and intra-class compactness, further sharpening the discriminative structure beyond what feature mimicry alone can produce under the student's reduced capacity. Experiments across different vision applications (i.e., urban scene parsing and medical image segmentation) show that SWARD achieves state-of-the-art performance.

Problem

• knowledge distillation
• cross-architectural
• semantic segmentation
• representational gap
• spatial dependencies

Innovation

• Stochastic Window Attention
• Cross-Architectural Distillation
• Semantic Segmentation
• Prototype Discriminative Regularization
• Knowledge Distillation