UNDREAM: Bridging Differentiable Rendering and Photorealistic Simulation for End-to-end Adversarial Attacks

2025-10-19

AI Summary

In safety-critical applications such as autonomous driving, the non-differentiability of photorealistic simulation environments hinders joint optimization of environmental factors (e.g., lighting, weather, viewpoint, object trajectories) for adversarial attacks. Method: This paper introduces the first end-to-end differentiable framework that unifies photorealistic simulation with differentiable rendering, enabling gradient-based co-optimization of multi-dimensional environmental variables. By tightly coupling physics engines, 3D scene modeling, and end-to-end adversarial perturbation generation, our approach synthesizes physically plausible and visually realistic 3D adversarial objects in simulation. Contribution/Results: Experiments demonstrate high attack success rates across diverse dynamic scenarios and strong generalization across varying sensor configurations and environmental conditions. The proposed method significantly enhances the fidelity and effectiveness of adversarial robustness evaluation for perception systems in realistic driving environments.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Deep learning models deployed in safety critical applications like autonomous driving use simulations to test their robustness against adversarial attacks in realistic conditions. However, these simulations are non-differentiable, forcing researchers to create attacks that do not integrate simulation environmental factors, reducing attack success. To address this limitation, we introduce UNDREAM, the first software framework that bridges the gap between photorealistic simulators and differentiable renderers to enable end-to-end optimization of adversarial perturbations on any 3D objects. UNDREAM enables manipulation of the environment by offering complete control over weather, lighting, backgrounds, camera angles, trajectories, and realistic human and object movements, thereby allowing the creation of diverse scenes. We showcase a wide array of distinct physically plausible adversarial objects that UNDREAM enables researchers to swiftly explore in different configurable environments. This combination of photorealistic simulation and differentiable optimization opens new avenues for advancing research of physical adversarial attacks.

Problem

• Bridging photorealistic simulation with differentiable rendering for attacks
• Enabling end-to-end adversarial perturbation optimization on 3D objects
• Creating physically plausible adversarial objects in configurable environments

Innovation

• Bridges photorealistic simulators with differentiable renderers
• Enables end-to-end optimization of adversarial perturbations
• Allows full control over environmental factors and movements