🤖 AI Summary
This study addresses the absence of standardized metrics for quantifying the distributional alignment between survey samples and target populations across multidimensional demographic characteristics. To this end, it introduces the Global Representativeness Index (GRI), which—by incorporating total variation distance into survey methodology for the first time—establishes a symmetric [0,1] scoring framework to assess the fidelity of samples with respect to complex demographic structures. The GRI leverages benchmark demographic data from the United Nations and Pew Research Center and complements design effect to form a novel paradigm for sample quality evaluation, implemented via an open-source Python library. Validation across multiple international survey datasets reveals that even large-scale probability samples typically achieve fine-grained GRI scores below 0.36, underscoring substantial deficiencies in current surveys’ demographic representativeness.
📝 Abstract
Global survey research increasingly informs high-stakes decisions in AI governance and cross-cultural policy, yet no standardized metric quantifies how well a sample's demographic composition matches its target population. Response rates and demographic quotas -- the prevailing proxies for sample quality -- measure effort and coverage but not distributional fidelity. This paper introduces the Global Representativeness Index (GRI), a framework grounded in Total Variation Distance that scores any survey sample against population benchmarks across multiple demographic dimensions on a [0, 1] scale. Validation on seven waves of the Global Dialogues survey (N = 7,500 across 60+ countries) finds fine-grained demographic GRI scores of only 0.33--0.36 -- roughly 43% of the theoretical maximum at that sample size. Cross-validation on the World Values Survey (seven waves, N = 403,000), Afrobarometer Round 9 (N = 53,000), and Latinobarometro (N = 19,000) reveals that even large probability surveys score below 0.22 on fine-grained global demographics when country coverage is limited. The GRI connects to classical survey statistics through the design effect; both metrics are recommended as a minimum summary of sample quality, since GRI quantifies demographic distance symmetrically while effective N captures the asymmetric inferential cost of underrepresentation. The framework is released as an open-source Python library with UN and Pew Research Center population benchmarks, applicable to survey research, machine learning dataset auditing, and AI evaluation benchmarks.