When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasters

2026-08-05

AI Summary

This work addresses systematic errors in frozen pretrained models that recurrently manifest in structured scenarios and resist correction through fine-tuning. To tackle this, the authors propose CRAFTER, a framework that, while keeping the backbone frozen, uniquely focuses feature engineering on modeling the failure process itself. CRAFTER automatically discovers interpretable corrective features by mining prediction residuals and employs them to drive a lightweight post-hoc corrector. It integrates compositional input channel search with large language model–generated named features, binary flags, and executable code, followed by gated unified validation for robust feature selection. A source-agnostic evaluation pipeline precisely attributes feature contributions. Evaluated across six datasets and six frozen backbones, CRAFTER nearly doubles the correction efficacy of existing methods, reduces error by up to 27% even for the weakest backbone, and demonstrates consistent robustness across diverse LLMs and fine-tuned models.

Abstract

Frozen pretrained forecasters often fail in structured, recurring ways that are costly to repair through fine-tuning. We study corrective feature discovery: mining interpretable features of a frozen forecaster's residual to drive a lightweight post-hoc corrector. Prior automated feature engineering models the data-generating process; corrective features instead model the model-failure process. We present CRAFTER (Corrective Residual Agent with Feature-based Temporal Exploration and Reasoning), which keeps the backbone frozen and mines its residual with two complementary generators: a compositional search over the raw input channels, and a large language model (LLM) that proposes named feature combinations, binary flags, and short executable code. A single validation-grounded gate accepts or rejects every candidate regardless of its origin, and a validation-selected corrector applies the accepted features or leaves the forecast unchanged. This source-agnostic pipeline also allows prior feature-engineering systems to be evaluated under identical conditions, making CRAFTER an instrument for attributing forecast improvements to the feature source alone. Across six public datasets and six frozen backbones, CRAFTER surpasses every dedicated feature-engineering system at every feature budget, roughly doubling the improvement achieved by the corrector alone and reducing the error of the weakest backbones by up to 27%. These gains are robust across different LLM backends and persist even when applied on top of fine-tuned backbones.

Problem

• corrective feature discovery
• black-box forecasters
• model failure
• post-hoc correction
• frozen models

Innovation

• corrective feature discovery
• frozen forecaster
• residual modeling
• large language model (LLM)
• post-hoc correction