Structured Secant Methods to Select Smoothing Parameters For General Smooth Models

2026-06-25

AI Summary

This work addresses the computational challenges of Bayesian marginal likelihood optimization in generalized smooth models, where traditional Laplace approximation relies on high-order derivatives that are both costly to compute and difficult to implement. The authors propose a quasi-Newton extended Fellner–Schall (qEFS) method that employs a structured limited-memory secant approximation to the Hessian of the log-likelihood, requiring only first-order derivatives to efficiently approximate critical subblocks. The approach further allows selective incorporation of exact Hessian columns to enhance accuracy. Balancing computational efficiency with estimation precision, qEFS demonstrates robustness under broader conditions. Empirical results show that qEFS converges to the classical EFS in simulations, substantially simplifies implementation in hidden Markov and Tweedie models, and achieves near-nominal performance in confidence interval coverage and model selection tasks.

Abstract

General smooth models replace parameters of a regular likelihood with additive models. The models can include parametric terms, Gaussian random effects, and smooth functions of covariates. The latter are parameterized via a reduced-rank spline basis and regularized via weighted quadratic penalties placed on the basis coefficients. Estimates for these weights (i.e., smoothing parameters) can be obtained by optimizing the Laplace-approximate Bayesian marginal likelihood. Existing (second-order) methods require the Hessian of the log-likelihood to solve this optimization problem approximately - exact optimization requires up to fourth order derivatives - which can be difficult to derive and expensive to evaluate. To address these problems, we present a quasi-Newton variant of the second-order Extended Fellner-Schall (EFS) optimization method. Our qEFS method relies on structured limited-memory secant approximations to the Hessian of the log-likelihood and is principally first-order. However, the approximation can also be accumulated for a sub-block of the Hessian, with the remaining columns being constrained to match those of the actual Hessian. The exact columns then provide additional structure for the sub-block approximation, which becomes more accurate as a result. We show that the qEFS method converges to the EFS method under certain conditions and continues to provide good estimates beyond these circumstances, which we illustrate in simulation studies. Secondary tasks involving the Hessian (confidence interval coverage & model selection) require partial approximations to achieve close to nominal performance. We provide Hidden Markov and Tweedie model examples, for which the qEFS method is substantially easier to implement than alternative methods.

Problem

• smoothing parameters
• general smooth models
• Laplace approximation
• Hessian matrix
• optimization

Innovation

• structured secant approximation
• smoothing parameter selection
• quasi-Newton method
• general smooth models
• Hessian approximation