Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks

2025-01-22

AI Summary

To address the scarcity and high cost of high-quality annotations in monocular depth estimation (MDE), this paper proposes a multi-source auxiliary task alternation training framework grounded in vision foundation models. The framework features a shared decoder, main-task-weighted loss, and—novelly—models semantic segmentation data as multi-label dense classification (MLDC) to serve as a strongly correlated auxiliary task. Experiments demonstrate that auxiliary tasks must be carefully designed rather than naively stacked. Our method achieves an average accuracy improvement of approximately 11% across mainstream MDE benchmarks. Moreover, it reduces annotation requirements by over 80% for equivalent performance and significantly enhances few-shot generalization. The source code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ~11%. Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.

Problem

• Monocular Depth Estimation
• Limited Annotated Data
• Auxiliary Tasks

Innovation

• Alternate Training Method
• Shared Decoder
• Monoocular Depth Estimation Enhancement