ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context

2024-03-04

Conference on Empirical Methods in Natural Language Processing

AI Summary

This work addresses three key challenges in table-sentence joint understanding: weak reasoning planning capability, poor generalization, and limited interpretability. We propose a “Plan-then-Reason” framework that first generates multi-step reasoning paths and then dynamically invokes either programmatic or textual submodules to execute each step. Our approach introduces a planning-driven, dual-modal (program + text) reasoning paradigm. To support training and evaluation, we construct TrixInstruct—a lightweight, high-generalization instruction-tuning dataset containing only 6K samples—and open-source the ProTrix model family. Experiments demonstrate substantial improvements in accuracy and explanation faithfulness across diverse table reasoning tasks, outperforming state-of-the-art prompting methods while reducing API calls and context dependency. All code, data, and models are publicly released to ensure full reproducibility.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Tables play a crucial role in conveying information in various domains. We propose a Plan-then-Reason framework to answer different types of user queries over tables with sentence context. The framework first plans the reasoning paths over the context, then assigns each step to program-based or textual reasoning to reach the final answer. This framework enhances the table reasoning abilities for both in-context learning and fine-tuning methods. GPT-3.5-Turbo following Plan-then-Reason framework surpasses other prompting baselines without self-consistency while using less API calls and in-context demonstrations. We also construct an instruction tuning set TrixInstruct to evaluate the effectiveness of fine-tuning with this framework. We present ProTrix model family by finetuning models on TrixInstruct. Our experiments show that ProTrix family generalizes to diverse unseen tabular tasks with only 6k training instances. We further demonstrate that ProTrix can generate accurate and faithful explanations to answer complex free-form questions. Our work underscores the importance of the planning and reasoning abilities towards a model over tabular tasks with generalizability and interpretability. We open-source our dataset and models at https://github.com/WilliamZR/ProTrix.

Problem

• Enhancing table reasoning with Plan-then-Reason framework
• Improving accuracy in answering diverse tabular queries
• Generalizing models for unseen tasks with minimal data

Innovation

• Plan-then-Reason framework for table queries
• Combines program-based and textual reasoning
• ProTrix model family with 6k training instances