Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space

2026-03-15

AI Summary

This work addresses the susceptibility of large language models to distractors in multiple-choice questions, which often leads to unstable preferences and answer oscillation. To mitigate this issue, the authors propose the Inclusion-of-Thoughts (IoT) method, which employs a self-filtering mechanism to assess the plausibility of candidate options and reformulates the question by explicitly removing distractors. This approach purifies the decision space, reduces cognitive load, and steers the model toward meaningful comparisons and chain-of-thought reasoning. IoT is the first method to explicitly alleviate preference instability caused by distractors, achieving substantial gains in reasoning performance across arithmetic, commonsense, and educational benchmarks while incurring minimal computational overhead. Additionally, it enhances the transparency and interpretability of model decisions.

Abstract

Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs). However, LLMs remain vulnerable to the presence of plausible distractors. This often diverts attention toward irrelevant choices, resulting in unstable oscillation between correct and incorrect answers. In this paper, we propose Inclusion-of-Thoughts (IoT), a progressive self-filtering strategy that is designed to mitigate this cognitive load (i.e., instability of model preferences under the presence of distractors) and enable the model to focus more effectively on plausible answers. Our method operates to reconstruct the MCQ using only plausible option choices, providing a controlled setting for examining comparative judgements and therefore the stability of the model's internal reasoning under perturbation. By explicitly documenting this filtering process, IoT also enhances the transparency and interpretability of the model's decision-making. Extensive empirical evaluation demonstrates that IoT substantially boosts chain-of-thought performance across a range of arithmetic, commonsense reasoning, and educational benchmarks with minimal computational overhead.

Problem

• preference instability
• distractors
• multiple-choice questions
• large language models
• cognitive load

Innovation

• Inclusion-of-Thoughts
• preference instability
• self-filtering
• distractor mitigation
• chain-of-thought reasoning