Learning the Pareto Frontier of Predictive Models under Distribution Shift

2026-08-01

AI Summary

This work addresses the challenge that under distribution shift, no single model reuse strategy—such as zero-shot inference, fine-tuning, or training from scratch—consistently achieves optimal performance, and existing approaches lack a unified framework to integrate heterogeneous pretrained models. To overcome this limitation, the authors propose Frontier Learning, a novel framework that unifies diverse reuse strategies into a single hypothesis class for the first time. It constructs a unified target-domain representation by concatenating internal features from white-box models with predictive outputs from black-box models, upon which a lightweight, regularized supervised learner is trained. Theoretically, this approach guarantees empirical risk no worse than any individual baseline strategy. Empirical results on DomainNet and VisDA for visual domain adaptation, as well as on MIMIC-IV-Notes for cross-ICU mortality prediction, demonstrate that Frontier Learning matches or surpasses the best single-strategy baseline, with particularly pronounced gains when no reliable baseline is available.

Abstract

Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can differ not only in performance but also in how they can be used: some only provide black-box predictions, while others may permit white-box access to internal representations that can be probed or fine-tuned. When deployed to the target domain in the presence of distribution shift, no single strategy, including zero-shot application, fine-tuning, or directly training a target-specific model, is uniformly the best. In this work, we propose Frontier Learning, a framework that treats a library of candidate models spanning different training histories and access regimes as complementary sources of information rather than mutually exclusive alternatives. Frontier Learning constructs a unified target-domain feature by concatenating internal representations from white-box candidates as well as prediction outputs from black-box candidates, then fits a lightweight, regularized supervised learner on this concatenated representation using labeled target data. Because the resulting hypothesis class contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases, empirical risk minimization over the frontier learner is guaranteed to be no worse, on the training sample, than any individual baseline. We evaluate the framework in simulations spanning varying degrees of source-target compatibility and in two real-world distribution-shift settings: visual domain adaptation on DomainNet/VisDA and clinical mortality prediction across intensive care unit domains using MIMIC-IV-Notes. Across all settings, Frontier Learning matches or outperforms the strongest individual reuse strategy, with the largest gains arising precisely when no single baseline is reliable across the range of shift considered.

Problem

• distribution shift
• pretrained models
• Pareto frontier
• model reuse
• domain adaptation

Innovation

• Frontier Learning
• distribution shift
• model reuse
• Pareto frontier
• feature concatenation