Exploring LLMs for Automated Pre-Testing of Cross-Cultural Surveys

2025-01-10

AI Summary

Cross-cultural questionnaire design in Information and Communication Technologies for Development (ICTD) is typically costly and time-intensive due to reliance on expert review and small-scale pilot testing. Method: This paper pioneers a systematic investigation of large language models (LLMs) for automating cross-cultural questionnaire pretesting. Starting from the U.S. Climate Opinion Survey, we employed LLM-driven text localization and cultural adaptation to generate a South Africa–contextualized version, which was evaluated alongside the direct translation in a controlled, dual-version experiment (N=116) on Prolific. Contribution/Results: The LLM-adapted version significantly outperformed the literal translation in comprehensibility and acceptability. Our work challenges the conventional expert- and pilot-dependent paradigm, empirically validating the feasibility and initial efficacy of LLM-assisted cross-cultural questionnaire design. It offers a scalable, low-cost, and efficient pathway for cultural adaptation in ICTD research.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLPCognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Designing culturally relevant questionnaires for ICTD research is challenging, particularly when adapting surveys for populations to non-western contexts. Prior work adapted questionnaires through expert reviews and pilot studies, which are resource-intensive and time-consuming. To address these challenges, we propose using large language models (LLMs) to automate the questionnaire pretesting process in cross-cultural settings. Our study used LLMs to adapt a U.S.-focused climate opinion survey for a South African audience. We then tested the adapted questionnaire with 116 South African participants via Prolific, asking them to provide feedback on both versions. Participants perceived the LLM-adapted questions as slightly more favorable than the traditional version. Our note opens discussions on the potential role of LLMs in adapting surveys and facilitating cross-cultural questionnaire design.

Problem

• Cultural Adaptation
• Questionnaire Design
• ICTD Research

Innovation

• Large Language Models
• Cross-cultural Adaptation
• Efficient Survey Design