Difference-Based Relational Learning for Zero-Shot Object-Goal Visual Navigation With Direct Sim-to-Real Transfer

2026-07-17

AI Summary

This work addresses the limited generalization of end-to-end deep reinforcement learning in zero-shot visual navigation due to the sim-to-real gap. To mitigate this issue, we propose the Temporal Difference Relational Network (T-DRN), which integrates a difference-based siamese feature extractor with a two-frame temporal buffering mechanism. This design enables the generation of domain-invariant relational representations while preserving object continuity under narrow field-of-view observations. Our approach is the first to incorporate difference-based relational learning and temporal modeling into an end-to-end navigation framework, effectively alleviating domain shift and enabling zero-shot transfer without fine-tuning. Extensive experiments demonstrate that T-DRN significantly outperforms strong baselines in the AI2-THOR simulation environment and exhibits robustness to real-world perceptual and actuation constraints when deployed on a physical wheeled robot.

Abstract

End-to-end deep reinforcement learning (DRL) for zero-shot object-goal visual navigation remains challenged by the sim-to-real gap, particularly variations in object appearance and restricted camera field-of-view (FoV). This letter proposes a Temporal Difference-Relational Network (T-DRN) for robust zero-shot sim-to-real transfer. T-DRN combines a Siamese difference-based feature extractor, which computes relational difference between the target and observed objects to produce domain-independent representations, with a dual-frame temporal buffer that preserves short-term object continuity under narrow FoV. Extensive experiments in AI2-THOR demonstrate that T-DRN improves zero-shot generalization in terms of success rates over strong baselines. Furthermore, T-DRN is systematically validated on a physical wheeled robot, demonstrating robust performance under real sensing and actuation constraints and supporting the feasibility of direct sim-to-real transfer.

Problem

• zero-shot navigation
• sim-to-real gap
• object appearance variation
• limited field-of-view
• visual navigation

Innovation

• Temporal Difference-Relational Network
• zero-shot visual navigation
• sim-to-real transfer
• difference-based relational learning
• domain-independent representation