HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators

2026-07-31

AI Summary

This work addresses the performance degradation of autoregressive neural operators in long-term prediction due to error accumulation and the lack of effective feedback from historical failure modes in existing training paradigms. The authors propose History-Enhanced Rollout Optimization (HERO), which, for the first time, leverages failure trajectories from the optimization history as relative supervisory signals. Building upon absolute trajectory supervision, HERO compares candidate trajectories generated by the current model, a lagged model, and perturbed inputs to identify the worst-performing failure trajectory, which is then used to construct a boundary target for sample-dependent gradient reweighting. Integrated with a backbone combining spectral methods and attention mechanisms, HERO substantially improves long-term prediction accuracy, rollout stability, and out-of-distribution robustness across nine PDE benchmarks, without incurring additional inference overhead.

Abstract

Neural operators provide fast surrogates for time-dependent partial differential equations (PDEs) by applying a learned evolution operator recursively to its own predictions, but this autoregressive rollout feeds every prediction error back as input, so local errors accumulate. Existing rollout-training strategies reduce the mismatch between training inputs and self-generated states, yet their supervision still measures only the absolute discrepancy from the ground-truth trajectory. Such supervision is therefore uninformative about whether the operator has overcome the long-horizon failure behaviors it exhibited earlier during optimization. We propose history-enriched rollout training (HERO), which augments conventional absolute trajectory supervision with relative supervision derived from the model's optimization history. HERO ranks detached candidate rollouts from a periodically refreshed lagged operator, the current model, and a perturbed input by rollout error, spectral discrepancy, energy drift, and error growth, and selects the strongest failure trajectory as reference. This reference enters a margin-based objective as a fixed comparison baseline, inducing a bounded, sample-dependent reweighting of the ground-truth rollout gradient rather than an independent gradient direction, which we further analyze theoretically. Experiments on nine PDE benchmarks with spectral and attention-based backbones show that HERO consistently improves long-horizon accuracy, stable rollout length, and out-of-distribution robustness at no inference-time cost. These results indicate that history-enriched relative supervision is effective for stabilizing long-horizon autoregressive prediction.

Problem

• autoregressive rollout
• error accumulation
• long-horizon prediction
• neural operators
• trajectory supervision

Innovation

• history-enriched supervision
• autoregressive neural operators
• rollout training
• relative supervision
• long-horizon prediction