SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

2026-06-08

AI Summary

This study addresses the challenge that large language models often struggle to disentangle semantic content from formal logical structure during reasoning, rendering them susceptible to content bias. To mitigate this issue, the authors propose the Syllogistic Evaluation Framework–Common Logic Grammar Construction (SEF-CLGC), which, for the first time, integrates formal logical symbols into small language models (SLMs). By employing a hybrid training approach that combines natural language with symbolic representations, the framework effectively decouples content-based and form-based reasoning processes. Experimental results on SemEval-2026 Task 11 Subtask 1 demonstrate that SEF-CLGC significantly reduces content bias while enhancing formal reasoning capabilities, achieving a content score of 27.80%.

Abstract

This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.

Problem

• formal reasoning
• content bias
• logical notation
• language model evaluation
• reasoning disentanglement

Innovation

• formal logical notation
• Small Language Models
• content bias reduction
• reasoning disentanglement
• symbolic-natural language integration