OmniPred: Language Models as Universal Regressors

2024-02-22

arXiv.org

AI Summary

Traditional regression models are task-specific and tightly coupled to domain-specific architectures, limiting generalization across heterogeneous numerical prediction tasks. Method: We propose an end-to-end textualized regression framework that encodes arbitrary numeric input–output pairs (x, y) as natural language sequences, fine-tunes pretrained Transformer language models on large-scale, multi-task regression data, and employs Google Vizier for data-driven black-box hyperparameter optimization—eliminating the need for task-specific architectures or numerical feature engineering. Contribution/Results: Our approach achieves state-of-the-art performance across diverse regression benchmarks, demonstrating for the first time that language models possess strong, generalizable capability for universal numerical prediction. It establishes a unified, text-only paradigm for regression, bypassing conventional numerical modeling assumptions and enabling cross-domain applicability without architectural modifications.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Classification and Regression
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Regression is a powerful tool to accurately predict the outcome metric of a system given a set of parameters, but has traditionally been restricted to methods which are only applicable to a specific task. In this paper, we propose OmniPred, a framework for training language models as universal end-to-end regressors over $(x,y)$ data from arbitrary formats. Using data sourced from Google Vizier, one of the largest proprietary blackbox optimization databases in the world, our extensive experiments demonstrate that language models are capable of very precise numerical regression using only textual representations of mathematical parameters and values, and if given the opportunity to train at scale over multiple tasks, can significantly outperform traditional regression models.

Problem

• Language Model
• Universal Prediction Tool
• Varied Prediction Tasks

Innovation

• OmniPred
• Language Model Adaptation
• Universal Prediction Tool