PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

2026-09-22

AI Summary

本文提出PotARCin基准，通过五个维度评估AI模型在抽象推理任务中的技能获取，以解决现有评估方法过于单一的问题。

Abstract

The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.

Problem

• Abstract Reasoning
• Skill Acquisition
• Evaluation Benchmark
• Diversity of Abilities
• ARC

Innovation

• Multi-Dimensional Evaluation
• Dynamic Generative Sampling
• Abstract Reasoning