🤖 AI Summary
This work addresses the computational burden and non-differentiability of traditional information criteria, which rely on discrete search procedures. We propose the Smooth Information Criterion (SIC), which employs an ε-scaling continuation strategy to continuously relax and progressively sharpen the model dimensionality penalty term. This reformulates discrete variable selection as a differentiable optimization problem, enabling coefficient-level selection in generalized linear models without requiring data-driven regularization parameters. Experimental results demonstrate that SIC replicates the model selection performance of exhaustive BIC search while outperforming stepwise regression and LASSO. Furthermore, it substantially reduces computational overhead in high-dimensional settings.
📝 Abstract
Variable selection using information criteria has an explicit statistical target but requires discrete search over candidate models. The smooth information criterion (SIC) replaces the discontinuous model-dimension term by a differentiable approximation, with an $ε$-telescoping continuation strategy progressively sharpening this approximation without data-driven selection of a regularisation-strength parameter. We develop SIC as a general procedure for coefficient-level variable selection in generalised linear models (GLMs) and use it to address a central question: how faithfully does smooth optimisation reproduce the corresponding discrete information-criterion selection problem? Focusing on BIC, we benchmark SIC directly against exhaustive subset selection where feasible, using exact support agreement, BIC difference and selection behaviour across a varying signal-strength boundary. Simulations in Gaussian, binomial and Poisson regression show that SIC closely reproduces exhaustive BIC selection and tracks the exact BIC selection boundary. Relative to stepwise BIC, LASSO, SCAD and MCP, SIC produces competitive variable-selection performance while retaining sparse models, with predictive performance broadly comparable across methods. Computational advantages over stepwise BIC increase with predictor dimension. In a real-data application with 16 candidate predictors, SIC recovers the globally BIC-optimal model among all 65,536 supports at a small fraction of the computational cost of exhaustive enumeration.