DPBridge: Latent Diffusion Bridge for Dense Prediction

2024-12-29

AI Summary

Diffusion models suffer from low accuracy, slow inference, and reliance on pure-noise initialization in dense image prediction tasks. To address these issues, we propose the Implicit Diffusion Bridge (IDB), which abandons conventional denoising trajectories and instead establishes an analytically tractable, end-to-end mapping from input images to dense output signal maps. IDB introduces the first fully solvable diffusion bridge mechanism, enabling deterministic implicit evolution from conditional inputs to target signals. Leveraging a pre-trained image diffusion backbone, we further design a lightweight visual prior fusion fine-tuning strategy that balances generalization capability and training stability. On semantic segmentation and depth estimation benchmarks, IDB achieves accuracy competitive with state-of-the-art feedforward and diffusion-based methods, while accelerating inference by an average factor of 3.2× and demonstrating improved training robustness.

Technology Category

Computer Vision: Diffusion Models for VisionSearch and Optimization: Learning to SearchMachine Learning: Deep Generative Models & Autoencoders

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models have demonstrated remarkable success in dense prediction problems, which aims to model per-pixel relationship between RGB images and dense signal maps, thanks to their ability to effectively capture complex data distributions. However, initiating the reverse sampling trajectory from uninformative noise prior introduces limitations such as degraded performance and slow inference speed. In this work, we propose DPBridge, a generative framework that formulates dense prediction tasks as image-conditioned generation problems and establishes a direct mapping between input image and its corresponding dense map based on fully-tractable diffusion bridge process. This approach addresses aforementioned limitations in conventional diffusion-based solutions. In addition, we introduce finetuning strategies to adapt our model from pretrained image diffusion backbone, leveraging its rich visual prior knowledge to facilitate both efficient training and robust generalization ability. Experimental results shows that our DPBridge can achieve competitive performance compared to both feed-forward and diffusion-based approaches across various benchmarks, highlighting its effectiveness and adaptability.

Problem

• Diffusion Models
• Image Signal Prediction
• Accuracy and Speed

Innovation

• DPBridge
• Image Prediction
• Diffusion Models