Random Forest-of-Thoughts: Uncertainty-aware Reasoning for Computational Social Science

2025-02-26

AI Summary

This paper addresses the limited uncertainty modeling capability of large language models (LLMs) in dynamic social survey analysis—stemming from their deterministic, single-path reasoning paradigm. To overcome this, we propose a novel random-forest-style thought generation mechanism. Our method constructs a diverse subspace of “thoughts,” integrating stochastic sampling, tree-based thought expansion, and ensemble response aggregation within a prompt engineering framework, thereby enabling scalable, uncertainty-aware, multi-path reasoning. Evaluated on two novel social survey analysis tasks—context-dependent option inference and dynamic attitude modeling—our approach significantly outperforms Chain-of-Thought and other baselines, achieving an average 12.7% improvement in reasoning accuracy. The core contribution is the first integration of an ensemble-based stochastic thinking mechanism into LLM reasoning, establishing a new paradigm for modeling uncertainty in complex social phenomena.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Social surveys in computational social science are well-designed by elaborate domain theories that can effectively reflect the interviewee's deep thoughts without concealing their true feelings. The candidate questionnaire options highly depend on the interviewee's previous answer, which results in the complexity of social survey analysis, the time, and the expertise required. The ability of large language models (LLMs) to perform complex reasoning is well-enhanced by prompting learning such as Chain-of-thought (CoT) but still confined to left-to-right decision-making processes or limited paths during inference. This means they can fall short in problems that require exploration and uncertainty searching. In response, a novel large language model prompting method, called Random Forest of Thoughts (RFoT), is proposed for generating uncertainty reasoning to fit the area of computational social science. The RFoT allows LLMs to perform deliberate decision-making by generating diverse thought space and randomly selecting the sub-thoughts to build the forest of thoughts. It can extend the exploration and prediction of overall performance, benefiting from the extensive research space of response. The method is applied to optimize computational social science analysis on two datasets covering a spectrum of social survey analysis problems. Our experiments show that RFoT significantly enhances language models' abilities on two novel social survey analysis problems requiring non-trivial reasoning.

Problem

• Enhance reasoning in social surveys
• Address uncertainty in LLM decision-making
• Optimize computational social science analysis

Innovation

• Random Forest of Thoughts
• Uncertainty-aware reasoning
• Diverse thought space exploration