RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models

2026-08-03

AI Summary

This work addresses the challenge in autonomous driving system development posed by the scarcity of paired video data capturing the same scene under diverse weather conditions, a limitation that hinders existing methods from simultaneously achieving photorealistic weather synthesis and high-fidelity preservation of scene structure. To overcome this, the authors propose RealWeather, a driving world model capable of high-fidelity, bidirectional long-tailed weather translation and zero-shot out-of-domain generalization without requiring paired training data. The approach innovatively integrates progressive realism bootstrapping, a pseudo-clear-weather generation pipeline, and reinforcement learning–based scene fidelity optimization to maintain geometric and semantic consistency while effectively suppressing visual hallucinations. Experimental results demonstrate that RealWeather significantly outperforms current state-of-the-art methods in both visual realism and structural fidelity.

Abstract

Realistic weather translation is valuable for developing and evaluating autonomous driving systems, yet collecting paired videos of the same scenes under different weather conditions at scale is impractical. Existing methods therefore rely on synthetic data, 3D weather editing, or geometry-conditioned generation, often compromising weather realism or scene fidelity. We propose RealWeather, a driving world model for both realistic and scene-faithful weather translation. Our key idea is to learn authentic weather dynamics directly from real-world videos. Specifically, RealWeather employs Progressive Realism Bootstrapping, an iterative data-refinement strategy. Assisted by an auxiliary Pseudo-Clear Generation pipeline, training initially starts with pseudo-style conditioning videos. As training proceeds, these inputs are progressively replaced with increasingly realistic videos generated by the model itself. This strategy bridges the pseudo-to-real domain gap, allowing the model to adapt seamlessly to real-world input distributions and naturally support bidirectional clear adverse translation. Furthermore, to strictly enforce structural integrity and suppress hallucinations, we introduce Scene-Fidelity RL Optimization, a reward-driven policy optimization strategy that explicitly penalizes alterations to safety-critical driving elements. Extensive experiments demonstrate that RealWeather significantly outperforms existing methods in visual realism and structural preservation, while enabling robust long-tail weather scenario generation and strong zero-shot out-of-distribution generalization.

Problem

• weather translation
• scene fidelity
• autonomous driving
• realistic synthesis
• domain gap

Innovation

• weather translation
• driving world model
• progressive realism bootstrapping
• scene-fidelity RL optimization
• zero-shot generalization