LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models

2025-04-18

AI Summary

Vision foundation models (e.g., DINOv2, CLIP) suffer from low feature resolution, limiting their applicability to pixel-level understanding tasks. To address this, we propose CoorFormer, a coordinate-driven cross-attention upsampler that introduces explicit 2D coordinate embeddings into the cross-attention mechanism—enabling resolution-agnostic feature reconstruction for the first time. Methodologically, we design a class-agnostic mask-guided self-distillation strategy that synthesizes high-fidelity pseudo-ground-truths from low-resolution features, eliminating reliance on annotated supervision. The framework is backbone-agnostic and plug-and-play, requiring no modifications to the pretrained model. Extensive experiments demonstrate that CoorFormer consistently outperforms existing upsampling approaches on downstream tasks—including semantic segmentation and depth estimation—producing sharper, more detail-rich high-resolution features. It achieves state-of-the-art performance across multiple benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Feature Construction/Reformulation
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved impressive results on various downstream tasks, but their limited feature resolution hampers performance in applications requiring pixel-level understanding. Feature upsampling offers a promising direction to address this challenge. In this work, we identify two critical factors for enhancing feature upsampling: the upsampler architecture and the training objective. For the upsampler architecture, we introduce a coordinate-based cross-attention transformer that integrates the high-resolution images with coordinates and low-resolution VFM features to generate sharp, high-quality features. For the training objective, we propose constructing high-resolution pseudo-groundtruth features by leveraging class-agnostic masks and self-distillation. Our approach effectively captures fine-grained details and adapts flexibly to various input and feature resolutions. Through experiments, we demonstrate that our approach significantly outperforms existing feature upsampling techniques across various downstream tasks. Our code is released at https://github.com/andrehuang/loftup.

Problem

• Enhancing feature resolution for pixel-level understanding in vision models
• Designing coordinate-based cross-attention transformer for sharp feature upsampling
• Creating high-resolution pseudo-groundtruth features via self-distillation and masks

Innovation

• Coordinate-based cross-attention transformer for upsampling
• High-resolution pseudo-groundtruth via class-agnostic masks
• Self-distillation training objective for feature enhancement