Large Language Models for Market Research: A Data-augmentation Approach

2024-12-26

AI Summary

Conjoint analysis in market research suffers from data scarcity, high survey costs, and systematic biases in traditional methods. Method: This paper proposes a statistical augmentation framework that integrates synthetically generated responses from large language models (LLMs) with a small set of real human responses, employing transfer learning to calibrate and mitigate LLMs’ inherent systematic preference biases. Unlike naive substitution approaches, we introduce the novel paradigm of “auxiliary LLM data”—where synthetic data serves solely to enhance estimation accuracy without replacing empirical observations. Contribution/Results: We theoretically establish consistency and asymptotic normality of the proposed estimator. Empirical evaluations on vaccine preference and sports car choice tasks demonstrate reduced estimation error versus conventional methods, cost savings of 24.9%–79.8% in data collection, and superior performance over baseline LLM substitution strategies—validating both efficacy and cross-domain robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have transformed artificial intelligence by excelling in complex natural language processing tasks. Their ability to generate human-like text has opened new possibilities for market research, particularly in conjoint analysis, where understanding consumer preferences is essential but often resource-intensive. Traditional survey-based methods face limitations in scalability and cost, making LLM-generated data a promising alternative. However, while LLMs have the potential to simulate real consumer behavior, recent studies highlight a significant gap between LLM-generated and human data, with biases introduced when substituting between the two. In this paper, we address this gap by proposing a novel statistical data augmentation approach that efficiently integrates LLM-generated data with real data in conjoint analysis. Our method leverages transfer learning principles to debias the LLM-generated data using a small amount of human data. This results in statistically robust estimators with consistent and asymptotically normal properties, in contrast to naive approaches that simply substitute human data with LLM-generated data, which can exacerbate bias. We validate our framework through an empirical study on COVID-19 vaccine preferences, demonstrating its superior ability to reduce estimation error and save data and costs by 24.9% to 79.8%. In contrast, naive approaches fail to save data due to the inherent biases in LLM-generated data compared to human data. Another empirical study on sports car choices validates the robustness of our results. Our findings suggest that while LLM-generated data is not a direct substitute for human responses, it can serve as a valuable complement when used within a robust statistical framework.

Problem

• Market Research
• Large Language Models
• Preference Analysis

Innovation

• Data Fusion Method
• Language Model Bias Correction
• Market Research Efficiency