Iterative Deepening Sampling for Large Language Models

2025-02-08

AI Summary

To address the limited self-assessment and self-correction capabilities of large language models (LLMs) in complex reasoning tasks, this paper proposes the Iterative Deepening Sampling (IDS) framework. IDS employs a human-triggered mechanism to jointly optimize intra-response and inter-response self-reflection. The method integrates heuristic sampling control, response re-scoring, multi-round self-verification, and backtracking-based correction, while jointly enforcing deterministic decoding constraints and uncertainty-aware resampling. Crucially, IDS operates without reliance on high-quality human feedback or reinforcement learning signals. Experimental results demonstrate substantial improvements in problem-solving success rates on the Math500 and AIME benchmarks, particularly for challenging problems. Ablation studies quantitatively validate the individual contributions of each component to reasoning depth and answer robustness.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The recent release of OpenAI's o1 models and other similar frameworks showcasing test-time scaling laws has demonstrated their exceptional capability to tackle complex reasoning tasks. Inspired by this, subsequent research has revealed that such test-time scaling laws hinge on the model's ability to search both within a single response (intra-response) and across multiple responses (inter-response) during training. Crucially, beyond selecting a single optimal response, the model must also develop robust self-correction capabilities within its own outputs. However, training models to achieve effective self-evaluation and self-correction remains a significant challenge, heavily dependent on the quality of self-reflection data. In this paper, we address this challenge by focusing on enhancing the quality of self-reflection data generation for complex problem-solving, which can subsequently improve the training of next-generation large language models (LLMs). Specifically, we explore how manually triggering a model's self-correction mechanisms can improve performance on challenging reasoning tasks. To this end, we propose a novel iterative deepening sampling algorithm framework designed to enhance self-correction and generate higher-quality samples. Through extensive experiments on Math500 and AIME benchmarks, we demonstrate that our method achieves a higher success rate on difficult tasks and provide detailed ablation studies to analyze its effectiveness across diverse settings.

Problem

• Enhancing self-reflection data quality
• Improving self-correction in LLMs
• Boosting complex reasoning task performance

Innovation

• Iterative deepening sampling algorithm
• Enhances self-correction mechanisms
• Improves quality of self-reflection data