EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision

2025-11-11

AI Summary

Addressing the challenging 2D–3D visual query localization problem in egocentric vision—complicated by camera motion, viewpoint shifts, and appearance variations—this paper proposes EAGLE, a unified framework. Its core innovation draws inspiration from avian memory consolidation, introducing a dual-path memory module: (i) a segmentation-guided, appearance-aware meta-learning memory that models long- and short-term object representations, and (ii) a tracking-driven, geometry-aware localization memory enabling rapid 3D back-projection. These pathways are synergistically fused via a vision-geometry Transformer. This design marks the first effort to jointly model and efficiently decouple appearance and geometric cues within a single memory architecture. Evaluated on the Ego4D-VQ benchmark, EAGLE achieves state-of-the-art performance in both retrieval accuracy and 3D spatial discrimination.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Localization, Mapping, and Navigation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-ofthe-art performance on the Ego4D-VQ benchmark.

Problem

• Addresses egocentric visual query localization challenges from camera motion and viewpoint changes
• Unifies 2D-3D localization through episodic appearance and geometry memory integration
• Enables precise contour delineation and accurate 3D back-projection in embodied AI systems

Innovation

• Episodic appearance- and geometry-aware memory framework
• Segmentation and tracking via structured memory banks
• Unified 2D-3D localization with visual geometry Transformer