Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning

2026-08-02

AI Summary

This work addresses the challenge of optimizing mathematical reasoning in large language models without relying on extensive human annotations or external reward models. The authors propose Cloud-ScPO, a novel framework that reveals, for the first time, that reasoning trajectories exhibit structured point-cloud distributions in latent space. Leveraging this insight, they introduce a topology-guided, component-wise soft k-nearest neighbor scoring mechanism that integrates cross-problem geometric structure with prompt-level self-consistency to enable semi-supervised preference optimization. Requiring only minimal annotated data, Cloud-ScPO outperforms ScPO by 4.49% on GSM8K and 4.19% on MATH-Numeric, effectively discriminating high-quality from low-quality reasoning trajectories while maintaining high correctness and reliability.

Abstract

Preference optimization improves mathematical reasoning in large language models (LLMs), but reliable chosen-rejected pairs usually require verified answers, human annotations, or external reward models. We investigate whether preference supervision can instead be derived from the model's internal representation geometry in a semi-supervised setting. Our analysis shows that reasoning trajectories generated across different mathematical problems form structured global point clouds in which correct and incorrect trajectories exhibit different geometric organization. Based on this observation, we propose Cloud--ScPO, a topology-guided preference-mining framework that uses a small labeled set to construct multiple correct and incorrect reference Clouds. Each trajectory is represented by a mean-pooled hidden state and scored against connectivity-induced components using a component-level soft $k$-nearest-neighbor measure averaged across reference banks. We combine this cross-problem Cloud signal with prompt-level self-consistency: self-consistency determines the answer-level preference direction, while Cloud scoring selects concrete trajectories and filters pairs by their score margin. Experiments on GSM8K and MATH-Numeric across four model settings show that Cloud--ScPO consistently improves over ScPO, with gains of up to 4.49\% on GSM8K and 4.19\% on MATH-Numeric. Pair-level analyses further show that Cloud--ScPO maintains comparable correctness reliability while more effectively separating informative chosen trajectories from incomplete, repetitive, or otherwise low-quality rejected responses.

Problem

• preference optimization
• semi-supervised learning
• large language models
• mathematical reasoning
• hidden-state geometry

Innovation

• preference optimization
• hidden-state geometry
• semi-supervised learning
• reasoning trajectories
• topology-guided mining