Open-Book Neural Algorithmic Reasoning

2024-12-30

AI Summary

Neural networks exhibit weak generalization on complex mathematical reasoning—particularly algorithmic reasoning—and struggle with cross-task knowledge transfer. Method: This paper proposes an open-book neural algorithmic reasoning framework that breaks from single-instance supervision by dynamically retrieving and integrating information from the entire training set during inference. It introduces a novel cross-task instance attention mechanism to model intrinsic relationships among the 30 algorithmic tasks in the CLRS benchmark, enabling interpretable multi-task collaborative training. Contribution/Results: Through attention-driven retrieval-aggregation and joint optimization, the model achieves significant improvements in reasoning accuracy on CLRS, empirically validating effective cross-task knowledge transfer. Additionally, it provides a task-relationship visualization tool, establishing a new paradigm for algorithm-level neural reasoning.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Neural algorithmic reasoning is an emerging area of machine learning that focuses on building neural networks capable of solving complex algorithmic tasks. Recent advancements predominantly follow the standard supervised learning paradigm -- feeding an individual problem instance into the network each time and training it to approximate the execution steps of a classical algorithm. We challenge this mode and propose a novel open-book learning framework. In this framework, whether during training or testing, the network can access and utilize all instances in the training dataset when reasoning for a given instance. Empirical evaluation is conducted on the challenging CLRS Algorithmic Reasoning Benchmark, which consists of 30 diverse algorithmic tasks. Our open-book learning framework exhibits a significant enhancement in neural reasoning capabilities. Further, we notice that there is recent literature suggesting that multi-task training on CLRS can improve the reasoning accuracy of certain tasks, implying intrinsic connections between different algorithmic tasks. We delve into this direction via the open-book framework. When the network reasons for a specific task, we enable it to aggregate information from training instances of other tasks in an attention-based manner. We show that this open-book attention mechanism offers insights into the inherent relationships among various tasks in the benchmark and provides a robust tool for interpretable multi-task training.

Problem

• Neural Networks
• Complex Mathematical Problems
• Performance Enhancement

Innovation

• Open Learning
• Attention Mechanism
• Problem Generalization