Distilling Answer Set Programming Theories from Large Language Models

2026-07-30

AI Summary

This work addresses the challenge of manually crafting answer set programming (ASP) theories, which is typically complex and time-consuming. The authors propose a neurosymbolic approach that leverages a large language model within a closed-loop environment featuring a fixed ASP solver. Starting from an empty initial file and using only a single, dataset-agnostic prompt, the method automatically distills complete and correct ASP theories from scratch within one hour, without any predefined rules. Evaluated on CLEVR, GQA, and CLEVRER benchmarks, the approach achieves end-to-end theory generation with state-of-the-art performance: 100% accuracy on CLEVR, 92.8%–98.8% on GQA, and 92.7%–95.3% on CLEVRER. The study further demonstrates, for the first time, the critical role of model scale in enabling complex logical reasoning, and publicly releases code, prompts, and distilled theories.

Abstract

Writing Answer Set Programming (ASP) theories from scratch is a difficult and time-consuming task. We take a neurosymbolic approach to study whether a model can distill complete and correct theories, given a fixed agent harness with the solver in the loop. The protocol is dataset-agnostic: with a single prompt and an empty file as the starting point the model is given a 1-hour time limit to derive a complete theory. We chose VQA as the application domain, three benchmarks (CLEVR, GQA, CLEVRER), as these are publicly available and non-trivial. In order to study the model scale required for solving this task we nine different models: four frontier (Claude Sonnet 4.6, Claude Opus 4.7, GPT-5, DeepSeek V4 Pro), two mid-tier (DeepSeek V4 Flash, gpt-oss-120b), and three open-weights (qwen3.6-27b, gpt-oss-20b, qwen3.5-9b). Three of four frontier models reach 100% on CLEVR and 92.8%-98.8% on GQA; on CLEVRER, Sonnet, Opus, DeepSeek V4 Pro score 92.7%-95.3%. GPT-5 reaches 98.7% on CLEVR but drops to 41.8% on GQA and to 86.7% on CLEVRER. Adding handwritten reference theories from other datasets moves the other three frontier models by at most +/-3.4 pp but reduces GPT-5's accuracy by 3-19 pp. We release the code, prompts, and theories distilled.

Problem

• Answer Set Programming
• theory distillation
• large language models
• neurosymbolic AI
• VQA

Innovation

• neurosymbolic
• Answer Set Programming
• theory distillation
• large language models
• visual question answering