Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve

2026-08-04

AI Summary

This study investigates how the rigid formatting and stylistic constraints inherent in traditional chain-of-thought (CoT) prompting may inadvertently hinder the core reasoning capabilities of large language models. Through systematic evaluation on mathematical reasoning benchmarks such as GSM8K, the authors compare zero-shot soft prompting against few-shot CoT across multiple medium-scale specialized and general-purpose models. They reveal a previously unobserved phenomenon: as model capacity increases, standard CoT prompting becomes a performance bottleneck, whereas lightweight soft prompting in a zero-shot setting consistently outperforms few-shot CoT—evidenced by an accuracy improvement from 77% to 84% on the Mathstral model, with similar gains observed across general models. These findings challenge the prevailing assumption of CoT’s universal efficacy and offer a new direction for designing more efficient reasoning prompts.

Abstract

Chain-of-Thought (CoT) prompting remains the standard baseline for evaluating models' reasoning abilities. Originally, this technique was introduced to elicit step-by-step reasoning from large language models (LLMs), which would otherwise tend to directly output the final answer. However, many modern LLMs produce CoT-style responses \textit{natively} when presented with reasoning tasks, which made us revisit the effectiveness of standard CoT prompting. We evaluate several modern mid-sized language models on a math problem-solving task and find that models specialized for reasoning achieve better performance in a simple zero-shot setting than when using few-shot CoT examples - significantly surpassing officially reported results at no additional cost (e.g., from $\sim$77\% to $\sim$84\% for Mathstral on GSM8K). For the tested general-purpose model, a zero-shot CoT prompt is also sufficient to outperform a few-shot CoT baseline. We attribute this to a `guidance-distraction' tradeoff: standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task. Our findings suggest that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger.

Problem

• Chain-of-Thought prompting
• large language models
• reasoning
• guidance-distraction tradeoff
• zero-shot

Innovation

• Chain-of-Thought prompting
• zero-shot reasoning
• guidance-distraction tradeoff
• large language models
• math problem solving