Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research

2026-09-23

AI Summary

论文提出了一种新的验证框架，用于解决合成研究中决策行为预测准确性的问题，并通过定义三个公正维度和子群体报告要求来提高验证的代表性和公平性。

Abstract

Researchers in industry and academia use synthetic survey respondents powered by large language models as substitutes for human samples. These synthetic populations require validation against real-world data, so researchers often address them using ad hoc comparisons with human surveys. Inspired by the intention-behaviour gap in behavioural science, we argue that these validations test the wrong thing for most applied cases where decision makers commission synthetic research to anticipate consequential behaviour. To address this problem, we propose a validation framework with two requirements. First, every validity claim must state its level of correspondence with human data: does the sample predict what the represented people do, which of four diagnostics (location, dispersion, response process and structure) does the validation address, and does the validation compare against experimental effects? Second, researchers must report validity claims for subgroups, since these groups are often the most affected by consequential decisions and aggregate accuracy hides their misrepresentation. Our validation framework operationalises three justice dimensions (distributional, procedural, and recognition) as measurable quantities and defines within-persona counterfactual experiments as a validation requirement. We then apply the framework to electric vehicle charging tariffs, before closing with a reporting checklist that researchers can use to make convincing validity claims.

Problem

• synthetic survey respondents
• validation framework
• consequential behaviour
• representational fairness
• subgroup validity

Innovation

• synthetic survey respondents
• validation framework
• justice dimensions
• counterfactual experiments