Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

2026-06-02

AI Summary

The geometric structure of hidden representations during language model reasoning remains poorly understood, limiting the effective elicitation of reasoning capabilities. This work reveals for the first time that input and reasoning-state embeddings exhibit high conicity and directional separation in representation space. Building on this observation, the authors formulate reasoning steering as an orthogonal rotation problem: a rotation matrix is estimated from a few correct examples, and a synthetic “thought vector” is injected at inference time to guide the model geometrically. The method requires no training, demonstrates strong generality, and improves accuracy in 30 out of 32 model–benchmark combinations across mathematical, scientific, and code-related tasks. Notably, it also achieves zero-shot generalization to MATH-Vision, a multimodal reasoning setting.

Abstract

Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer. Yet the internal structure of representation space when reasoning remains poorly understood: how do a model's hidden representations differ during thinking versus the embeddings of the input prompt, and can this structure be exploited to elicit stronger reasoning at inference time? We show that both input embeddings and thinking embeddings (mean-pooled last-layer hidden states over the prompt and reasoning trace, respectively) exhibit extremely high conicity, with all vectors clustering tightly around a single mean direction. Crucially, these mean input and thinking directions are non-collinear, with thinking embeddings occupying a geometrically distinct region of embedding space across many different models and benchmark tasks. This observation motivates casting the input-to-thinking transition as a rotation problem admitting a closed-form solution via orthogonal Procrustes analysis. We propose Rotate2Think, a training-free method that estimates this rotation from a small set of correctly solved examples and injects the resulting synthetic thinking vector between thinking delimiters at inference time, providing a geometric primer at the onset of the reasoning trace. Evaluated across multiple benchmarks and model families, Rotate2Think improves accuracy in 30 of 32 model-benchmark configurations across mathematics, science, and code tasks, and generalizes zero-shot to multimodal reasoning on MATH-Vision.

Problem

• reasoning
• representation space
• embedding geometry
• language models
• thinking embeddings

Innovation

• orthogonal rotation
• geometric priming
• reasoning traces
• embedding conicity
• training-free inference