The Impact of Question Framing on the Precision of Automatic Occupation Coding

2025-01-09

AI Summary

This study investigates how question wording in occupational surveys affects the accuracy and linguistic variability of automated job classification. Using a multi-wave German survey experiment, we compare the coding performance of tools including CASCOT and OccuCoDe under two question formats: “job title” versus “occupational tasks,” while quantifying response-level linguistic diversity. Our key contribution is the first empirical demonstration that question format critically moderates automated coding outcomes: the “job title” formulation significantly improves both coding efficiency and accuracy, whereas adding task examples—though enriching response detail—induces lexical homogenization, reducing coding accuracy by 12–18%. These findings uncover a fundamental design-driven mechanism influencing automated occupational classification, establishing that questionnaire structure directly shapes the quality of coded occupational data. The results provide rigorous empirical evidence and methodological guidance for optimizing occupational data collection and coding practices in large-scale social and labor market research.

Technology Category

Natural Language Processing: Question AnsweringApplication Domains: Humanities & Computational Social ScienceHumans and AI: Crowd Sourcing and Human Computation

Application Category

Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasetsWeb Mining and Content Analysis: Community question answeringSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Occupational data play a vital role in research, official statistics, and policymaking, yet their collection and accurate classification remain a persistent challenge. This study investigates the effects of occupational question wording on data variability and the performance of automatic coding tools. Through a series of survey experiments conducted and replicated in Germany, we tested two widely-used occupational question formats: one focusing on"job title"(Berufsbezeichnung) and another on"occupational tasks"(berufliche T""atigkeit). Our analysis reveals that automatic coding tools, such as CASCOT and OccuCoDe, exhibit significant sensitivity to the form and origin of the data. Specifically, these tools performed more efficiently when coding responses to the job title question format compared to the occupational task format. Additionally, we found that including examples of main tasks and duties in the questions led respondents to provide more detailed but less linguistically diverse responses. This reduced diversity may negatively affect the precision of automatic coding. These findings highlight the importance of tailoring automatic coding tools to the specific structure and origin of the data they are applied to. We emphasize the need for further research to optimize question design and coding tools for greater accuracy and applicability in occupational data collection.

Problem

• Investigates how question wording affects occupational data coding accuracy
• Compares job title versus task-based question formats for coding efficiency
• Highlights need for harmonized survey questions and robust coding tools

Innovation

• Used split-ballot survey experiment
• Compared job title vs task formats
• Tested automatic coding tools sensitivity