"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models

2024-09-27

arXiv.org

AI Summary

To address the challenge of constructing interpretable models for few-shot tabular data, this paper proposes Zero-shot Decision Trees (ZDT): a framework that generates structurally sound and semantically coherent decision trees—without any training data or fine-tuning—by leveraging the implicit world knowledge encoded in large language models (LLMs) via prompt engineering, and subsequently extracts topological embeddings from the generated trees. The method comprises symbolic parsing of tree structures and learnable tree embedding encoding. To our knowledge, this is the first work achieving pure zero-shot decision tree generation and embedding learning. Experiments on multiple small-scale tabular benchmarks demonstrate that ZDT achieves significantly higher accuracy than conventional data-driven decision trees. Moreover, its learned embeddings consistently outperform those derived from XGBoost and other tree-based models in downstream tasks, establishing a novel, knowledge-driven, and highly interpretable paradigm for low-resource scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can even surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform better than data-driven tree-based embeddings on average. Our decision tree induction and embedding approaches can therefore serve as new knowledge-driven baselines for data-driven machine learning methods in the low-data regime. Furthermore, they offer ways to harness the rich world knowledge within LLMs for tabular machine learning tasks. Our code and results are available at https://github.com/ml-lab-htw/llm-trees.

Problem

• Zero-shot decision tree induction using LLMs without training data
• Comparing zero-shot trees with data-driven trees on tabular datasets
• Enhancing tabular ML tasks via LLM knowledge-driven embeddings

Innovation

• Zero-shot decision tree induction using LLMs
• Knowledge-driven embeddings outperform data-driven ones
• LLMs generate interpretable models without training data