Think before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering

2024-01-03

IEEE Workshop/Winter Conference on Applications of Computer Vision

AI Summary

Neural-symbolic models exhibit insufficient causal reasoning capabilities for counterfactual question answering (CQA) in videos. Method: We propose a neuro-symbolic framework jointly driven by an explicit symbolic causal graph and Answer Set Programming (ASP). Specifically: (i) we construct a symbolic causal graph to explicitly encode inter-event causal structures; (ii) we pioneer ASP-based dynamic scheduling of perception modules and language-model-driven dynamical simulation; (iii) we employ GPT-3.5/GPT-4 as promptable, interpretable counterfactual simulators. Contribution/Results: This work is the first to use ASP for logical orchestration of neural computation pipelines, endowing the model with strong causal interpretability and counterfactual controllability. Our method achieves state-of-the-art performance on CLEVRER and significantly improves counterfactual QA accuracy on CRAFT via causal-graph-guided prompting.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Causal and temporal reasoning about video dynamics is a challenging problem. While neuro-symbolic models that combine symbolic reasoning with neural-based perception and prediction have shown promise, they exhibit limitations, especially in answering counterfactual questions. This paper introduces a method to enhance a neuro-symbolic model for counterfactual reasoning, leveraging symbolic reasoning about causal relations among events. We define the notion of a causal graph to represent such relations and use Answer Set Programming (ASP), a declarative logic programming method, to find how to coordinate perception and simulation modules. We validate the effectiveness of our approach on two benchmarks, CLEVRER and CRAFT. Our enhancement achieves state-of-the-art performance on the CLEVRER challenge, significantly outperforming existing models. In the case of the CRAFT benchmark, we leverage a large pre-trained language model, such as GPT-3.5 and GPT-4, as a proxy for a dynamics simulator. Our findings show that this method can further improve its performance on counterfactual questions by providing alternative prompts instructed by symbolic causal reasoning.

Problem

• Enhancing neuro-symbolic models for counterfactual reasoning in videos
• Using symbolic causal graphs to coordinate perception and simulation
• Improving performance on counterfactual questions with causal reasoning

Innovation

• Symbolic reasoning orchestrates neural computation
• Answer Set Programming coordinates perception modules
• Pre-trained language models enhance counterfactual performance