A Semiparametric Nonlinear Mixed Effects Model with Penalized Splines Using Automatic Differentiation

2026-03-12

AI Summary

This study addresses the challenge of jointly modeling population-level trajectories and individual heterogeneity in nonlinear mixed-effects models by proposing an efficient estimation approach that integrates penalized splines with subject-specific transformations. The population trajectory is represented via penalized splines, while Laplace approximation is employed to handle integrals involving random effects. Leveraging automatic differentiation within the Template Model Builder (TMB) framework, the method enables joint optimization of smoothing parameters and variance components, achieving substantial computational gains without compromising statistical accuracy. Simulation studies demonstrate superior performance over existing methods, and the approach is successfully applied to longitudinal data on height growth in infants during the first two years after birth.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present an estimation procedure for nonlinear mixed-effects models in which the population trajectory is represented by penalized splines and adapted to individuals via subject-specific transformation parameters. By exploiting the mixed model representation of penalized splines, the level of smoothness can be estimated jointly with other variance components. The integration over random effects needed to obtain the marginal likelihood is carried out using the Laplace approximation. Exact derivatives for evaluation and maximization of the resulting likelihood are obtained via automatic differentiation implemented through Template Model Builder. In simulation studies, the method produces improved inferential performance and reduced computational burden when compared to the existing procedure. The approach is further illustrated through a case study on infant height growth in the first two years of life.

Problem

• nonlinear mixed-effects model
• penalized splines
• automatic differentiation
• Laplace approximation
• growth trajectory

Innovation

• penalized splines
• nonlinear mixed-effects model
• automatic differentiation
• Laplace approximation
• semiparametric modeling