From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

2026-04-07

AI Summary

This study systematically investigates the impact of numerical feature preprocessing on tabular deep learning performance, with a focus on spline-based encoding for continuous variables. The authors propose a differentiable knot parameterization method that enables end-to-end joint optimization of knot locations for B-splines, M-splines, and I-splines alongside backbone architectures such as MLPs, ResNets, and FT-Transformers. They empirically compare uniform, quantile-based, target-aware, and learnable knot placement strategies across classification and regression tasks. Results show that piecewise linear encoding yields the most robust performance in classification settings, while regression accuracy is highly sensitive to the choice of spline type, knot strategy, and output dimensionality. Although learnable knots can enhance performance, they substantially increase training overhead.

Abstract

Numerical preprocessing remains an important component of tabular deep learning, where the representation of continuous features can strongly affect downstream performance. Although its importance is well established for classical statistical and machine learning models, the role of explicit numerical preprocessing in tabular deep learning remains less well understood. In this work, we study this question with a focus on spline-based numerical encodings. We investigate three spline families for encoding numerical features, namely B-splines, M-splines, and integrated splines (I-splines), under uniform, quantile-based, target-aware, and learnable-knot placement. For the learnable-knot variants, we use a differentiable knot parameterization that enables stable end-to-end optimization of knot locations jointly with the backbone. We evaluate these encodings on a diverse collection of public regression and classification datasets using MLP, ResNet, and FT-Transformer backbones, and compare them against common numerical preprocessing baselines. Our results show that the effect of numerical encodings depends strongly on the task, output size, and backbone. For classification, piecewise-linear encoding (PLE) is the most robust choice overall, while spline-based encodings remain competitive. For regression, no single encoding dominates uniformly. Instead, performance depends on the spline family, knot-placement strategy, and output size, with larger gains typically observed for MLP and ResNet than for FT-Transformer. We further find that learnable-knot variants can be optimized stably under the proposed parameterization, but may substantially increase training cost, especially for M-spline and I-spline expansions. Overall, the results show that numerical encodings should be assessed not only in terms of predictive performance, but also in terms of computational overhead.

Problem

• numerical preprocessing
• tabular deep learning
• spline-based encodings
• feature representation
• continuous features

Innovation

• learnable knots
• spline-based encoding
• differentiable parameterization
• tabular deep learning
• numerical preprocessing