MotherTree: Meta-learning on synthetic data improves decision tree training

2026-10-07

AI Summary

This study addresses the limitations of traditional decision trees lacking meta-learning priors and the difficulty of foundation models in generating auditable, standalone models. To this end, we propose MotherTree, which employs a tabular Transformer architecture pretrained on synthetic data to directly output hard axis-aligned decision trees in a single forward pass. Furthermore, it introduces a novel meta-learning paradigm that eliminates the need for reference tree supervision, effectively transforming black-box predictions into interpretable, independently deployable traditional decision trees. Experimental results demonstrate that MotherTree achieves performance comparable to classical algorithms on standard benchmarks while significantly outperforming gradient-based learning from scratch. Additionally, it serves as a strong initializer that enhances fine-tuning efficacy, successfully balancing model performance with transparency.

Abstract

Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially in small-sample regimes. However, this approach does not produce a standalone model that can be inspected in isolation. We introduce MotherTree, a tabular transformer that meta-learns decision tree induction: given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. MotherTree is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. On established benchmarks with controlled sample size, the approach is competitive with size-matched trees from common algorithms: recursive partitioning, gradient-based tree learning, globally optimal trees, and distillation from tabular foundation models. Notably, MotherTree consistently improves over from-scratch gradient-based learning and acts as a strong initializer: task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. These results show that meta-learning can provide effective inductive biases for learning stand-alone, small decision tree classifiers.

Problem

• decision tree
• meta-learning
• tabular foundation models
• synthetic data
• small-sample regimes

Innovation

• Meta-learning
• Decision Tree
• Synthetic Data
• Tabular Transformer
• Inductive Bias