Optimal Estimation of Two-Way Effects under Limited Mobility

2025-06-27

AI Summary

This paper addresses the problem of optimal estimation of bilateral effects—such as teacher–student matching effects—in settings with limited mobility, modeled as weakly connected bipartite graphs. We propose a novel empirical Bayes method that integrates asymptotic analysis of small eigenvalues of the graph Laplacian with empirical Bayes shrinkage, achieving asymptotically optimal estimation under compound loss. The method employs unbiased risk estimation (UBRE) for fully automatic, data-driven hyperparameter selection. It exhibits strong robustness to prior misspecification and degradation in graph connectivity. Theoretically, we establish consistency and asymptotic normality even under weak connectivity assumptions. Empirically, the approach successfully estimates teacher value-added effects using real-world teacher–student linkage data from North Carolina, demonstrating both statistical efficacy and practical applicability.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Reasoning under Uncertainty: Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We propose an empirical Bayes estimator for two-way effects in linked data sets based on a novel prior that leverages patterns of assortative matching observed in the data. To capture limited mobility we model the bipartite graph associated with the matched data in an asymptotic framework where its Laplacian matrix has small eigenvalues that converge to zero. The prior hyperparameters that control the shrinkage are determined by minimizing an unbiased risk estimate. We show the proposed empirical Bayes estimator is asymptotically optimal in compound loss, despite the weak connectivity of the bipartite graph and the potential misspecification of the prior. We estimate teacher values-added from a linked North Carolina Education Research Data Center student-teacher data set.

Problem

• Estimating two-way effects in linked data sets
• Modeling limited mobility in bipartite graphs
• Optimizing shrinkage hyperparameters for empirical Bayes

Innovation

• Empirical Bayes estimator for two-way effects
• Model bipartite graph with small eigenvalues
• Unbiased risk estimate for hyperparameters