Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?

2026-08-05

AI Summary

This study investigates whether language models can perform reasoning about necessity and possibility according to specified modal logic semantics rather than default logical systems. By constructing paired problems with identical syntactic formulas but differing semantic conditions—such as accessibility relations or domains—and automatically verifying that their labels are logically opposite, the work isolates the models’ reliance on syntax versus semantics while holding logical form constant. Integrating formal modal logic, automated theorem proving, and countermodel generation, the authors evaluate major language models under both direct prompting and reasoning-augmented settings. Results show that all four models perform below random chance under direct prompting; however, when reasoning mode is enabled, DeepSeek V4 Flash’s accuracy surges from 4.4% to 88.1%, underscoring the critical role of explicit reasoning in adhering to prescribed semantic constraints.

Abstract

Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.

Problem

• modal logic
• language models
• semantic reasoning
• frame conditions
• domain assumptions

Innovation

• modal logic
• semantic evaluation
• reasoning mode
• frame conditions
• language models