BundleFit: Display and See-Through Models for Augmented Reality Head-Mounted Displays

2024-12-30

AI Summary

Current display-through augmented reality (AR) head-mounted displays (HMDs) face a threefold bottleneck in joint optical modeling: low accuracy (due to black-box models), poor hardware adaptability (in white-box models), and insufficient real-time performance. To address these challenges, we propose BundleFit—a novel “external ray-bundle fitting” paradigm. Rather than modeling internal optical elements, BundleFit treats the entire optical system as a black box and performs lightweight gradient-based optimization solely on incident and emergent ray bundles at the input and output interfaces. It jointly leverages simulation and physical device calibration to estimate external geometric parameters. The method supports warm-start initialization, achieving a favorable trade-off among modeling fidelity, runtime efficiency (with bounded latency), and deployment simplicity. Experimental evaluation on real-world AR HMDs demonstrates significant improvements in display-through optical alignment accuracy, confirming its practical viability and commercial readiness.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Computer Vision: Learning & Optimization for CV
• Machine Learning: Hardware-aware ML

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The head-mounted display is a vital component of augmented reality, incorporating optics with complex display and see-through optical behavior. Computationally modeling these optical behaviors requires meeting three key criteria: accuracy, efficiency, and accessibility. In recent years, various approaches have been proposed to model display and see-through optics, which can broadly be classified into black-box and white-box models. However, both categories face significant limitations that hinder their adoption in commercial applications. To overcome these challenges, we leveraged prior knowledge of ray bundle properties outside the optical hardware and proposed a novel bundle-fit-based model. In this approach, the ray paths within the optics are treated as a black box, while a lightweight optimization problem is solved to fit the ray bundle outside the optics. This method effectively addresses the accuracy issues of black-box models and the accessibility challenges of white-box models. Although our model involves runtime optimization, this is typically not a concern, as it can use the solution from a previous query to initialize the optimization for the current query. We evaluated the performance of our proposed method through both simulations and experiments on real hardware, demonstrating its effectiveness.

Problem

• Augmented Reality
• Head-Mounted Displays
• Model Improvement

Innovation

• BundleFit
• Black-box and White-box Integration
• Augmented Reality Head-mounted Display