Jeffreys-Type Penalized GEE for Correlated Binary Data with an Odds-Ratio Parameterization

2026-06-14

AI Summary

This study addresses the instability of conventional generalized estimating equations (GEE) in separation scenarios—such as small samples, sparse data, or rare events—where non-convergence or extreme estimates commonly occur, particularly under non-independent working correlation structures. To overcome these limitations, the authors propose a penalized GEE framework that integrates Jeffreys’ prior penalty with a marginalized odds ratio parameterization, effectively circumventing the breakdown of traditional correlation parameters at extreme probabilities and guaranteeing finite estimates. The approach accommodates multiple link functions, including logit, probit, clog-log, and cauchit, and introduces both a one-step algorithm (OPGEE) and a hybrid variant (HPGEE) to enhance computational efficiency. Simulation studies and an analysis of a respiratory disease trial demonstrate that the proposed method substantially outperforms standard GEE under separation while maintaining comparable performance in regular settings. The methodology is implemented in the R package geer.

Abstract

Generalized estimating equations (GEE) are widely used for population-averaged inference on correlated binary responses, but ordinary GEE can fail under separation, a situation that is more likely in small-sample, sparse, or rare-event settings, leading to nonconvergence, infinite or extreme estimates, and unreliable inference. Existing penalized GEE (PGEE) approaches mitigate some of these problems but do not generally guarantee finite estimates under nonindependence working structures and often rely on correlation-coefficient parameterizations whose admissible range shrinks as fitted probabilities approach zero or one, forcing the working association toward independence under separation. We propose a PGEE framework that combines a Jeffreys-prior penalty with marginalized odds-ratio working parameterizations. The odds-ratio parameterization avoids this failure, while the penalty, with tunable strength $δ$ and default $δ= 1/2$, stabilizes estimation under separation. Under working independence, PGEE reduces to the Jeffreys-prior penalized maximum-likelihood estimator, yielding finite estimates for logit, probit, complementary log-log, and cauchit links. Under nonindependence odds-ratio structures, where a formal finiteness guarantee is unavailable, PGEE achieves near-complete empirical convergence even in separated settings. We also propose one-step and hybrid variants, OPGEE and HPGEE, that reduce computational cost. Simulations show that all three variants substantially outperform ordinary GEE under separation while retaining the performance of ordinary GEE in regular settings. We illustrate the method using a respiratory-illness trial in which ordinary GEE fails, and provide an implementation in the R package geer.

Problem

• separation
• correlated binary data
• generalized estimating equations
• nonconvergence
• finite estimation

Innovation

• Jeffreys prior
• penalized GEE
• odds-ratio parameterization
• separation
• correlated binary data