Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

2026-10-02

AI Summary

This study addresses the high acquisition cost and redundancy of multimodal features at test time, as well as the difficulty of modality selection when downstream tasks are unknown. To tackle these challenges, this work proposes ECHO-k, a self-supervised framework that leverages pretrained representations as proxy objectives and integrates reinforcement learning to achieve task-agnostic sequential modality selection. The primary contribution is the first task-agnostic, self-supervised paradigm for test-time feature acquisition, accompanied by theoretical guarantees in linear settings and a cost-aware deployment pathway. By synergizing deep pretraining, self-supervision, and reinforcement learning strategies, ECHO-k significantly enhances downstream performance under budget constraints across diverse foundation model backends, consistently outperforming existing baselines.

Abstract

Recent progress in multimodal, high-dimensional learning has enabled foundation models to process heterogeneous, large-scale data. However, at test time, acquiring all features or modalities can be prohibitively costly and often redundant. Sequentially selecting informative modalities is therefore critical, yet challenging when the downstream task or prediction target is unknown. To this end, we introduce ECHO-$k$, a task-agnostic and self-supervised learning principle for modality acquisition: we use a deep model's internal pretrained representations (e.g., from a foundation model) as proxy targets that summarize cross-modal information. We provide theoretical guarantees in a stylized linear setting that motivate a reinforcement learning (RL) policy for sequential modality selection. Across task-agnostic and label-free acquisition baselines, ECHO-$k$ consistently improves budgeted downstream performance across diverse foundation-model backends. Our method provides a principled route to cost-aware test-time deployment, with implications for any multimodal system where measurements are expensive or time-constrained, and downstream tasks unknown a priori.

Problem

• modality acquisition
• self-supervised learning
• test-time deployment
• multimodal systems
• task-agnostic

Innovation

• Self-Supervised Learning
• Test-Time Modality Acquisition
• Reinforcement Learning
• Foundation Models
• Task-Agnostic