StaticSegFormer: An Efficient High-Performance Semantic Segmentation Based on Static Structured Pruning

2026-08-05

AI Summary

This work addresses the inefficiency of existing dynamic structured pruning methods for semantic segmentation models, which, despite reducing computational cost, yield suboptimal inference speed on GPUs. To overcome this limitation, the authors propose a static structured pruning approach tailored specifically for attention layers within the SegFormer architecture, achieving compression without any degradation in mean Intersection-over-Union (mIoU). Notably, this method is the first to substantially improve GPU inference frame rates on high-resolution images while preserving accuracy, making it particularly suitable for scenarios involving compact encoders and large input images. Evaluated on the Cityscapes dataset, the approach demonstrates a 34% relative speedup in inference latency alongside a significant reduction in FLOPs, confirming its practicality and efficiency.

Abstract

Structured pruning enhances the efficiency of deep neural networks (DNNs) by eliminating groups of parameters during inference. Previous methods mostly reduce computational complexity (FLOPs), while semantic segmentation performance (mIoU) slightly drops. Accordingly, recent dynamic structured pruning methods aim at reducing the performance drop, while lowering the FLOPs even more. However, on the ADE20K and Cityscapes benchmarks, our study reveals that on a GPU platform such dynamic methods exhibit a surprisingly low frame rate far below a simple static approach, while having comparable results in mIoU and FLOPs. To address this issue, we propose a static structured pruning method for attention layers, that achieves both, a lower FLOPs and a high frame rate [fps] of the SegFormer network, the latter increased by up to 34% relative on the Cityscapes dataset, while having no mIoU performance drop at all. Our so-called StaticSegFormer method is strongest for small encoders and large images.

Problem

• semantic segmentation
• structured pruning
• frame rate
• computational efficiency
• performance drop

Innovation

• static structured pruning
• semantic segmentation
• SegFormer
• computational efficiency
• frame rate