Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

2026-06-01

AI Summary

This study addresses a critical limitation in current large language models (LLMs), which often generate hypotheses that overlook researcher-specified covariates and are thus susceptible to confounding, hindering the identification of genuine linguistic differences within specific subpopulations. To remedy this, the paper introduces, for the first time, a conditional hypothesis generation framework that explicitly incorporates covariates into the LLM-based hypothesis formulation process. The approach detects sign reversals through feature–covariate interaction terms and mitigates inter-group imbalance via within-layer mean centering and inverse frequency weighting. By integrating econometric principles with LLM capabilities, the proposed method substantially outperforms global baselines in both synthetic experiments and expert evaluations, yielding hypotheses that are not only tailored to particular subgroups but also more interpretable and practically actionable.

Abstract

A core goal of computational social science is to discover interpretable differences in how language varies across outcomes of interest, such as political affiliation or instructional quality. Recent LLM-based hypothesis generation methods describe such differences in natural language, but select for globally discriminative patterns without accounting for covariates that shape the data based on researchers' domain knowledge. When covariates are ignored, selected patterns can reflect confounds rather than differences of substantive interest. We introduce conditional hypothesis generation, a framework that incorporates researcher-specified covariates to steer hypothesis discovery toward differences that hold within relevant subgroups. Two challenges arise: the target subgroup may be underrepresented (stratum imbalance), and the direction of a difference may reverse across subgroups (sign reversal). We propose two econometrics-inspired methods: one introduces feature--covariate interactions to detect sign reversals, and the other applies within-stratum demeaning and inverse-frequency reweighting to equalize underrepresented strata. Synthetic experiments show each method outperforms global baselines in its targeted setting, and expert evaluation on two real-world datasets confirms that covariate-aware generation surfaces more useful hypotheses within relevant subgroups.

Problem

• conditional hypothesis generation
• covariates
• confounding
• subgroup analysis
• computational social science

Innovation

• conditional hypothesis generation
• covariate-aware analysis
• feature-covariate interaction
• within-stratum demeaning
• inverse-frequency reweighting