FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution

2026-09-26

AI Summary

This study addresses the poor generalization of depth super-resolution under unknown distributions and the training instability caused by heterogeneous data. To this end, it proposes a cross-domain reconstruction framework integrating RGB-guided 2D Gaussian Splatting with heterogeneous federated learning. Methodologically, we introduce the first RGB-guided anisotropic Gaussian deformation strategy to achieve high-fidelity depth modeling, combined with a heterogeneous federated learning mechanism that optimizes large-scale foundation models while effectively mitigating data distribution shifts. Experimental results demonstrate that the proposed approach comprehensively outperforms state-of-the-art methods in zero-shot evaluation, exhibiting superior scene generalization capability and robustness to noise.

Abstract

We introduce FoundDSR, a generalizable foundation model for robust depth reconstruction across unseen data distributions using RGB-D pairs. FoundDSR begins with a guided 2D Gaussian Splatting strategy to model depth representations with Gaussian primitives. This strategy employs high-resolution RGB as prompts to optimize the Gaussian parameters, thereby encouraging each Gaussian primitive to anisotropically deform along high-frequency structural directions. The resulting Gaussian-upsampled representations are then mapped to high-resolution depth through an effective depth reconstruction branch. Furthermore, to mitigate training instability and bias toward dominant sources caused by distribution gaps in large-scale heterogeneous data, we introduce heterogeneous federated learning that allocates each data source to an independent client for local optimization and global aggregation. This design effectively endows FoundDSR with stable scalability to diverse and large-scale training data. Extensive zero-shot evaluations on synthetic, real-world, arbitrary-scale, and noisy conditions demonstrate that FoundDSR consistently outperforms existing state-of-the-art approaches, confirming its strong robustness and generalization to unknown scenes.

Problem

• Depth Super-Resolution
• Generalization
• Heterogeneous Data
• Zero-shot
• Robustness

Innovation

• Depth Super-Resolution
• 2D Gaussian Splatting
• Foundation Model
• Federated Learning
• Zero-shot Generalization