Machine Learning for the Production of Official Statistics: Density Ratio Estimation using Biased Transaction Data for Japanese labor statistics

2025-10-28

AI Summary

This paper addresses the significant time lag—up to one year—in official labor statistics, which rely on traditional surveys and thus impede timely economic policymaking. To overcome this limitation, we propose a novel paradigm leveraging non-representative point-of-sale (POS) transaction data from private Japanese employment agencies. Methodologically, we introduce density ratio estimation—a technique previously unused in official statistics—within a supervised learning framework designed for covariate shift, enabling systematic correction of representativeness bias in non-probability data sources. Our key contribution is breaking the temporal bottleneck of survey-based sampling, enabling near-real-time estimation of critical labor market indicators, including job vacancy rates and recruitment activity. Empirical results demonstrate that our approach reduces statistical release cycles from annual to daily granularity, markedly enhancing policy responsiveness and data-driven decision-making capacity.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Intelligent Robots: State Estimation
• Philosophy and Ethics of AI: AI & Jobs/Labor

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

National statistical institutes are beginning to use non-traditional data sources to produce official statistics. These sources, originally collected for non-statistical purposes, include point-of-sales(POS) data and mobile phone global positioning system(GPS) data. Such data have the potential to significantly enhance the usefulness of official statistics. In the era of big data, many private companies are accumulating vast amounts of transaction data. Exploring how to leverage these data for official statistics is increasingly important. However, progress has been slower than expected, mainly because such data are not collected through sample-based survey methods and therefore exhibit substantial selection bias. If this bias can be properly addressed, these data could become a valuable resource for official statistics, substantially expanding their scope and improving the quality of decision-making, including economic policy. This paper demonstrates that even biased transaction data can be useful for producing official statistics for prompt release, by drawing on the concepts of density ratio estimation and supervised learning under covariate shift, both developed in the field of machine learning. As a case study, we show that preliminary statistics can be produced in a timely manner using biased data from a Japanese private employment agency. This approach enables the early release of a key labor market indicator that would otherwise be delayed by up to a year, thereby making it unavailable for timely decision-making.

Problem

• Addressing selection bias in non-traditional data for official statistics
• Using machine learning to correct biased transaction data
• Producing timely labor statistics from biased private employment data

Innovation

• Density ratio estimation corrects biased transaction data
• Supervised learning addresses covariate shift in statistics
• Machine learning enables timely labor indicator release