DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction

2026-07-17

AI Summary

This work addresses the computational redundancy and negative transfer between tasks that hinder the joint optimization of semantic segmentation and depth estimation in robotic perception when using vision foundation models (VFMs). To overcome these limitations, the authors propose DPNeXt, a lightweight multi-scale feature fusion decoder framework designed as a drop-in replacement for the standard DPT decoder. Key innovations include a dual-depthwise separable inverted residual bottleneck architecture that drastically reduces parameter count, a task-decoupling module that enhances the utilization efficiency of frozen VFMs, and a symmetric multi-task boundary guidance (MTBG) mechanism that enforces geometric consistency and mitigates negative transfer without requiring additional annotations. Experiments demonstrate state-of-the-art performance on both Cityscapes and NYUv2: DPNeXt-S reduces trainable parameters by 78.6% while achieving the fastest inference speed, and DPNeXt-B attains superior accuracy with fewer parameters.

Abstract

Multi-Task Learning (MTL) in robotics perception systems supports comprehensive 3D spatial scene understanding by integrating semantic segmentation and depth estimation. While Vision Foundation Models (VFMs) are increasingly adopted as robust feature encoders, existing decoding strategies present a critical bottleneck. To address this, we propose DPNeXt, a streamlined multi-scale feature fusion decoder and efficient alternative to the standard Dense Prediction Transformer (DPT). DPNeXt uses dual depthwise separable inverted bottlenecks to improve frozen VFM utilization through fusion-centric decoding and independent task modularization. To further mitigate negative inductive transfer between tasks, we introduce the Multi-Task Boundary Guidance (MTBG) strategy. Unlike prior boundary-aware methods that add fusion modules or gating, MTBG applies symmetric boundary-focused supervision to encourage geometric consistency without extra annotation or inference cost. Experiments on Cityscapes show that DPNeXt-S outperforms prior state-of-the-art (SOTA) MTL models, while DPNeXt-B further improves the overall performance and achieves the best results among the compared methods. On NYUv2, DPNeXt-B also achieves the best semantic segmentation and depth estimation results among the compared methods while requiring substantially fewer trainable parameters than prior large-scale MTL models. Compared with the standard DPT, DPNeXt-S reduces trainable parameters by 78.6% and achieves the fastest inference speed among the compared models on resource-constrained laptop hardware. The source code, model checkpoints, and a demo video will be made available at https://github.com/kangjehun/DPNeXt.

Problem

• Multi-Task Learning
• Dense Prediction
• Vision Transformer
• Feature Fusion
• Efficient Decoding

Innovation

• DPNeXt
• multi-scale feature fusion
• multi-task learning
• boundary guidance
• lightweight decoder