ALICE: In-context, Zero-shot, Mutual Information Estimation

2026-09-28

AI Summary

This study addresses the limitations of existing mutual information estimators, which require retraining and are constrained by data types in small-sample, cross-distribution scenarios. We propose a foundation model based on rectified flow velocity fields that computes mutual information by integrating differences between joint and conditional velocity fields via a fixed integral identity. Pretrained exclusively on synthetic distributions, the model enables zero-shot in-context estimation. For the first time, a single architecture achieves the accuracy of specialized models on unseen data while natively supporting multi-dimensional heterogeneous data and variable-length samples. Evaluations on standard benchmarks and applications in biology, genetics, and neuroscience demonstrate that our approach delivers high-precision, cross-domain, zero-shot mutual information analysis.

Abstract

Estimating mutual information (MI) from samples is a central objective in a variety of scientific fields. Modern neural estimators are accurate in the large-data regime, but they fall short when data is scarce, and each must be fit anew for every distribution under study. Current estimators are moreover tied to specific data types. These constraints limit their adoption in many applications where per-distribution training is impractical and sample sizes are small. We present ALICE, a foundation model that removes per-distribution training, while achieving competitive estimation accuracy. Trained exclusively on a broad family of synthetic distributions, ALICE acts as an in-context estimator of rectified-flow velocity fields: conditioned on samples of an unseen distribution, it estimates that distribution's velocity field without any explicit training. MI is then obtained through a fixed identity that integrates the squared difference between the joint and conditional fields. We validate ALICE on a standard, challenging benchmark and apply it in three domains, biology, genetics, and neuroscience, whose data the model has never seen. For the first time, we show that a single model closes the gap with neural estimators trained separately for each distribution, while natively supporting different data dimensionality and sample cardinality, enabling zero-shot MI analysis across scientific domains.

Problem

• Mutual Information Estimation
• Zero-shot Learning
• Data Scarcity
• Foundation Model

Innovation

• Mutual Information Estimation
• Foundation Model
• In-context Learning
• Zero-shot
• Rectified Flow