Reframing preprocessing selection as model-internal calibration in near-infrared spectroscopy: A large-scale benchmark of operator-adaptive PLS and Ridge models

2026-05-13

AI Summary

This study addresses the instability, poor auditability, and sensitivity to small sample sizes inherent in traditional near-infrared spectroscopic modeling, which relies on external preprocessing searches. To overcome these limitations, the authors propose an Operator-Adaptive Calibration framework (AOM) that embeds common preprocessing techniques—such as Standard Normal Variate (SNV), Multiplicative Scatter Correction (MSC), and Asymmetric Least Squares (ASLS)—as learnable linear operators integrated within local ensemble branches, thereby preventing information leakage. This approach unifies preprocessing and model training while preserving the interpretability of wavelength coefficients. Built upon PLS and Ridge regression, AOM leverages efficient algorithms and dual kernel formulations to enable rapid training and exact coefficient recovery. Evaluated across more than 50 heterogeneous datasets, AOM-PLS achieves a median RMSEP reduction of 4% over conventional PLS (outperforming it in 42 cases), while AOM-Ridge yields an average improvement of 2.22% (winning in 35 cases), with training times of only a few seconds.

Abstract

Near-infrared spectroscopy (NIRS) is rapid and non-destructive, but reliable calibration still depends heavily on spectral preprocessing. In routine practice, preprocessing is often selected by large external pipeline searches that are costly, unstable on small calibration sets, and difficult to audit. We introduce operator-adaptive calibration, a framework that moves linear preprocessing selection inside the calibration model. Candidate treatments are encoded as linear spectral operators, while nonlinear or sample-adaptive corrections such as SNV, MSC, and ASLS are handled as fold-local branches to prevent leakage. We instantiate the framework for PLS and Ridge regression. For PLS, covariance identities enable fast NIPALS and SIMPLS variants while preserving original-wavelength coefficients. For Ridge, operator-adaptive kernels yield a dual formulation with recoverable original-space coefficients. The approach was evaluated on more than 50 heterogeneous NIRS datasets against conventional PLS, Ridge, CatBoost, and CNN baselines under documented search budgets. Compact operator-adaptive PLS with ASLS branch preprocessing achieved a median RMSEP/PLS ratio of 0.960 with 42 wins on 57 datasets, while a deployable AOM-Ridge selector improved over tuned Ridge by a median 2.22% with 35 wins on 52 datasets. The proposed models reduce dependence on large preprocessing-HPO campaigns, produce traceable operator choices, retain interpretable coefficients, and fit in seconds for compact AOM-PLS. Operator-adaptive calibration therefore offers a practical route to faster, more robust, and more auditable NIRS method development.

Problem

• near-infrared spectroscopy
• spectral preprocessing
• calibration
• hyperparameter optimization
• model interpretability

Innovation

• operator-adaptive calibration
• spectral preprocessing
• PLS regression
• Ridge regression
• near-infrared spectroscopy