MatrixFormer: A Foundation Model for Matrix Completion

2026-10-05

AI Summary

This study addresses the limitation of existing matrix completion models that often neglect the two-dimensional structure of data, thereby constraining the efficiency of missing value prediction. To this end, we propose a native Transformer architecture that innovates the modeling paradigm by treating the matrix as a holistic entity. The method designs a pretraining strategy grounded in low-rank assumptions and latent factor matrices, leveraging synthetic data for pretraining. It directly outputs distributional predictions for all missing entries in a single forward pass, thereby achieving zero-shot generalization capabilities. Experimental results demonstrate that this unified framework attains competitive performance across diverse tasks, including causal inference, tabular data imputation, and recommender systems. Ultimately, this work establishes an efficient and unified new paradigm for matrix completion.

Abstract

Matrix completion underlies problems from tabular imputation to causal inference, yet existing tabular foundation models treat it as entry-by-entry prediction, repeating context for every target and discarding the matrix's two-dimensional structure. We introduce MatrixFormer, a pre-trained matrix-native transformer that predicts a full distribution for every missing entry in a single forward pass. MatrixFormer is trained entirely on synthetic low-rank and latent-factor matrices under diverse missingness patterns. Applied zero-shot and with the same model weights, MatrixFormer achieves competitive performance on causal inference panel-data tasks, language-model benchmark-score completion, tabular imputation, and recommendation systems matrix completion. These results position MatrixFormer as a general-purpose foundation model for matrix completion.

Problem

• Matrix Completion
• Tabular Imputation
• Foundation Model
• Causal Inference

Innovation

• Matrix Completion
• Foundation Model
• Matrix-Native Transformer
• Zero-Shot Generalization
• Synthetic Pre-training