🤖 AI Summary
To address the challenge of robustly selecting the regularization parameter in Eilers–Whittaker smoothing under large-scale data and serially correlated noise, this paper proposes an automatic optimization method based on residual spectral entropy. The core innovation lies in modeling the sigmoidal relationship between residual spectral entropy and the regularization parameter, with the optimal parameter identified at the absolute maximum of this S-shaped curve. Unlike cross-validation and the V-curve method, the proposed approach is insensitive to noise structure, computationally efficient, and highly robust. Leveraging spectral entropy analysis, S-curve modeling, and Euclidean distance–assisted decision-making, extensive validation on diverse synthetic and real-world time-series datasets demonstrates substantial improvements in both parameter selection stability and smoothing accuracy. The method establishes a new, interpretable, reproducible, and scalable paradigm for automated hyperparameter tuning of Eilers–Whittaker smoothers.
📝 Abstract
The Eilers-Whittaker method for data smoothing effectiveness depends on the choice of the regularisation parameter, and automatic selection is a necessity for large datasets. Common methods, such as leave-one-out cross-validation, can perform poorly when serially correlated noise is present. We propose a novel procedure for selecting the control parameter based on the spectral entropy of the residuals. We define an S-curve from the Euclidean distance between points in a plot of the spectral entropy of the residuals versus that of the smoothed signal. The regularisation parameter corresponding to the absolute maximum of this S-curve is chosen as the optimal parameter. Using simulated data, we benchmarked our method against cross-validation and the V-curve. Validation was also performed on diverse experimental data. This robust and straightforward procedure can be a valuable addition to the available selection methods for the Eilers smoother.