ArchitectureIQ: On the Measure of Training Intuition

2026-09-30

AI Summary

This study addresses the limited structured reasoning capabilities and perceptual blind spots regarding data attributes in current large language models (LLMs). To this end, it proposes the ArchitectureIQ benchmark, which leverages synthetic data construction, multi-model comparison, and chain-of-thought analysis to systematically quantify, for the first time, the intuitive differences between LLMs and humans in predicting optimal training recipes, while also exploring knowledge base compression techniques. The findings reveal that frontier models achieve approximately 76% accuracy, outperforming human experts, yet treat "data" as dark matter within AI comprehension. Furthermore, the work demonstrates that merely twenty structured knowledge items enable GPT-4o to approximate the performance of Claude Opus 5, thereby facilitating efficient enhancement of weaker models.

Abstract

Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuition of LLMs and humans, we introduce the ArchitectureIQ benchmark. Each question presents a synthetic dataset and several training recipes, and the test-taker is asked to predict the recipe yielding the best test metric. Overall, we find that LLMs' model intuition is good but has four limitations: (1) The intuition is imperfect, or even sub-human in some cases. Frontier models achieve around 76% accuracy (random choice 33%) vs best human researcher (66.0%), yet remain far from perfect. For architecture-only questions, best human achieves 65% while GPT-6 Astra only has 38%. (2) The intuition is empirical, not structured, supported by the fact that more CoT compute does not lead to substantial improvement. Unlike math, we still lack a "Science of AI" language that enables structured reasoning on AI. (3) The intuition is not maximally condensed, and can be further compressed into a knoledge base. Our constructed knowledge base with only 20 items yields large gains for weak models: GPT-4o equipped with the accumulated knowledge almost matches the performance of Claude Opus 5. (4) The intuition is insensitive to dataset properties, but the best model should in general depend on data properties. This suggests that data is the real "dark matter" in AI -- LLMs (so do human researchers) understand too little about data, even less than model architectures.

Innovation

• ArchitectureIQ benchmark
• training intuition
• knowledge base compression
• structured reasoning
• data properties