ACT: Agentic Classification Tree

2025-09-30

AI Summary

In high-stakes domains, AI decision-making requires transparency, explainability, and auditability—yet conventional decision trees are limited to structured data, while large language models (LLMs), though effective on unstructured text, lack verifiable and traceable reasoning. Method: This paper introduces the first method for constructing Natural Language Decision Trees (NL-Trees) over unstructured text. It reformulates decision-tree splitting as an optimizable natural language problem, integrating LLM-generated feedback, TextGrad-based gradient optimization, and information-entropy-driven impurity evaluation to iteratively induce interpretable, auditable classification paths. Contribution/Results: NL-Trees achieve classification accuracy comparable to or exceeding state-of-the-art prompt-engineering approaches on benchmark text classification tasks. Crucially, they produce explicit, hierarchical, and human-verifiable reasoning chains—thereby bridging the long-standing gap between interpretability and robust reasoning over unstructured textual data.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Transparent, Interpretable, Explainable ML
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

When used in high-stakes settings, AI systems are expected to produce decisions that are transparent, interpretable, and auditable, a requirement increasingly expected by regulations. Decision trees such as CART provide clear and verifiable rules, but they are restricted to structured tabular data and cannot operate directly on unstructured inputs such as text. In practice, large language models (LLMs) are widely used for such data, yet prompting strategies such as chain-of-thought or prompt optimization still rely on free-form reasoning, limiting their ability to ensure trustworthy behaviors. We present the Agentic Classification Tree (ACT), which extends decision-tree methodology to unstructured inputs by formulating each split as a natural-language question, refined through impurity-based evaluation and LLM feedback via TextGrad. Experiments on text benchmarks show that ACT matches or surpasses prompting-based baselines while producing transparent and interpretable decision paths.

Problem

• Extends decision trees to handle unstructured text inputs
• Ensures transparent and interpretable AI decision paths
• Replaces free-form reasoning with verifiable natural-language questions

Innovation

• Extends decision trees to unstructured data via natural-language splits
• Refines splits using impurity evaluation and LLM feedback
• Produces transparent interpretable decision paths matching baseline performance