feature engineering

Modeling heterogeneous, asynchronous, high-dimensional, and mixed-structured/unstructured clinical data in intensive care units (ICUs) remains a fundamental challenge. To address this, we propose ICU-BERT—the first Transformer-based pretraining framework specifically designed for critical care settings. Our method introduces (1) a multi-task self-supervised pretraining paradigm coupled with a multi-token asynchronous input strategy to accommodate irregular temporal dynamics, and (2) integration of dense semantic embeddings from biomedical large language models, enabling end-to-end, feature-engineering-free robust representation learning. Trained on MIMIC-IV, ICU-BERT achieves state-of-the-art performance across five diverse downstream clinical tasks—including mortality risk prediction and ICD coding—as well as on four independent external ICU datasets. Results demonstrate substantial improvements in generalizability and clinical utility of AI-driven decision support systems, validating its effectiveness in real-world critical care applications.

This study addresses the automatic extraction of physiologically or clinically meaningful features from intracranial pressure (ICP) waveforms to enhance diagnostic capabilities. Leveraging ICP data from 60 patients, the authors first segmented the signals into individual cardiac cycles and then developed a convolutional neural network to classify seven distinct body positions. Notably, they introduced a neural attention mechanism to localize critical regions within the ICP waveform—an approach that not only successfully identified waveform segments significantly associated with different body positions but also established a scalable, data-driven framework for feature discovery. This work provides a novel pathway toward deeper physiological interpretation of ICP dynamics and improved clinical decision-making.

Existing data pruning methods suffer from complex designs and poorly understood mechanisms of their key components, hindering progress in the field. Method: This paper pioneers a decoupled framework that separates data pruning into two orthogonal modules—“data representation” and “selection algorithm”—and systematically evaluates their independent contributions to instance selection efficacy in NLP model training. Through theoretical analysis and extensive empirical evaluation across multiple benchmarks (including gradient- and embedding-based representations and diverse selection algorithms), we assess their relative impact. Contribution/Results: We find that representation quality dominates selection algorithm choice: high-fidelity representations (e.g., training gradients) substantially improve pruning effectiveness, whereas no selection algorithm exhibits consistent superiority across tasks; even for identical objectives, different algorithms yield markedly divergent selected subsets. This work establishes an interpretable, reusable analytical framework for data pruning and identifies representation optimization—not algorithmic refinement—as the primary lever for enhancing pruning efficiency.

This work addresses the limitations of handcrafted input representations in supervised learning on complex heterogeneous data—such as time series, text, and structured records—by proposing a large language model (LLM)-based agent pipeline. The approach automatically derives global normalization rules from a small set of diverse textual serialization examples and combines them with task-conditioned local rules to construct efficient, auditable, and low-cost tabular input representations. Evaluated on 15 clinical tasks in the EHRSHOT benchmark, the method significantly outperforms conventional count-based feature models, naive LLM text serialization baselines, and large-scale pretrained clinical foundation models, demonstrating strong effectiveness and generalization capability in few-shot medical settings.

Existing approaches to modeling medical tabular data often neglect column-level contextual information (e.g., header semantics), require labor-intensive, manual preprocessing, and lack semantic interpretability. Method: We propose TabText—a novel framework that systematically encodes tabular structure into promptable natural language text, enabling end-to-end semantic representation learning via large language models (LLMs). TabText integrates structure-aware prompt engineering with a multi-task health prediction fine-tuning paradigm, supporting zero- or light-preprocessing modeling while remaining compatible with conventional feature fusion. Results: Evaluated on nine clinical prediction tasks, TabText alone establishes high-performance, lightweight baselines. When fused with traditional features, it yields an average AUC improvement of 6%, with even the worst-case gain reaching 6%, significantly enhancing model robustness and generalization across diverse healthcare scenarios.

This work proposes a context-aware multi-model optimization approach to overcome the limitations of relying on a single optimal model. By systematically identifying multiple models that exhibit substantially different feature selections yet achieve comparable predictive performance, the method preserves overall accuracy while enhancing interpretability. Applied to the METABRIC gene expression dataset, the approach successfully generates a diverse ensemble of high-performing models, uncovering multiple plausible biological interpretation pathways underlying the data. Compared to baseline methods, the resulting models demonstrate superior trade-offs between feature dissimilarity and performance consistency, thereby significantly improving both model interpretability and scientific insight.

This work investigates the optimal choice of representation dimensionality to achieve maximal generalization performance in settings where pretraining and downstream task data exhibit significant imbalance. By modeling pretraining as principal component analysis on unlabeled data and downstream learning as linear regression on labeled data, the authors employ high-dimensional statistical analysis to derive, for the first time, exact expressions for training and generalization errors in the high-dimensional asymptotic limit. Their theoretical results establish a quantitative relationship among representation dimensionality, data scale, and task alignment, yielding an analytical condition for the optimal representation size: low-dimensional compressed representations are preferable when labeled downstream data are scarce but pretraining data are abundant, whereas high-dimensional representations are advantageous in the opposite regime. The study further quantifies the extent to which unlabeled data can substitute for labeled samples and validates the proposed mechanism in both autoencoders and large language models.

This study addresses the challenge that ICU time-series prediction models trained on large hospitals often fail to generalize to resource-constrained smaller facilities. To tackle this, the authors frame cross-institutional model transfer as a domain-incremental continual learning task, enabling adaptation to the target domain’s data distribution while preserving knowledge from the source domain. They establish the first benchmark for domain-incremental continual learning on ICU time-series data and systematically evaluate representative methods—such as experience replay and Elastic Weight Consolidation (EWC)—using real-world multicenter datasets to assess their capacity for knowledge retention and transfer. Their analysis reveals substantial differences in measurement distributions and frequencies across ICUs in distinct U.S. regions, thereby demonstrating both the promise and limitations of current continual learning approaches in clinical settings.

This study systematically investigates the mechanisms by which data scale, model complexity, and input modality influence the generalization performance of vision models. Within a unified experimental framework, the authors conduct controlled and large-scale ablation studies on synthetic functions and the CIFAR dataset, employing polynomial fitting, diverse CNN and Transformer architectures, and multimodal inputs—including RGB, grayscale, gradients, edges, and wavelet representations—to quantitatively compare the effects of these three core factors for the first time. The findings reveal that increasing training data consistently enhances generalization; greater model complexity yields non-monotonic improvements; removing color information substantially degrades performance; and the efficacy of explicit handcrafted priors is highly dependent on model architecture.