Bayesian Estimation of Variance under Fine Stratification via Mean-Variance Smoothing

2026-03-03

AI Summary

This study addresses the challenge of variance estimation in fine-stratified sampling when stratum sample sizes are extremely small—often as low as one unit—where conventional estimators fail and common pooling strategies introduce positive bias, leading to overly wide confidence intervals. The authors propose a novel Bayesian variance estimation approach that avoids stratum pooling altogether. Their method introduces, for the first time in this context, a joint nonparametric smoothing mechanism for stratum means and variances, implemented via penalized splines to enable robust inference. Simulation studies and an application to data from the National Survey of Family Growth (NSFG) demonstrate that the proposed estimator substantially reduces bias and outperforms the method of Breidt et al. (2016), yielding more accurate variance estimates and more efficient confidence intervals.

Technology Category

• Machine Learning: Bayesian Learning
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Probabilistic Inference

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Fine stratification survey is useful in many applications as its point estimator is unbiased, but the variance estimator under the design cannot be easily obtained, particularly when the sample size per stratum is as small as one unit. One common practice to overcome this difficulty is to collapse strata in pairs to create pseudo-strata and then estimate the variance. The estimator of variance achieved is not design-unbiased, and the positive bias increases as the population means of the paired pseudo-strata become more variant. The resulting confidence intervals can be unnecessarily large. In this paper, we propose a new Bayesian estimator for variance which does not rely on collapsing strata, unlike the previous methods given in the literature. We employ the penalized spline method for smoothing the mean and variance together in a nonparametric way. Furthermore, we make comparisons with the earlier work of Breidt et al. (2016). Throughout multiple simulation studies and an illustration using data from the National Survey of Family Growth (NSFG), we demonstrate the favorable performance of our methodology.

Problem

• fine stratification
• variance estimation
• small sample size
• design-unbiasedness
• confidence interval

Innovation

• Bayesian variance estimation
• fine stratification
• penalized splines
• mean-variance smoothing
• nonparametric smoothing