Agentic Relative Camera Pose Estimation via Learned Ranking and Verification

2026-09-29

AI Summary

This study addresses the challenge that single camera pose estimators struggle with complex scenarios involving wide baselines and textureless regions. To overcome this limitation, we propose PoseAgent, a multi-agent framework that introduces learnable ranking and verification mechanisms. By employing profiling, ranking, and verification agents, PoseAgent dynamically orchestrates multiple candidate estimators to adaptively schedule the optimal solution. The core innovation lies in constructing a pose error verification network that surpasses existing models, enabling precise estimator evaluation and efficient collaboration. Experimental results demonstrate that PoseAgent achieves up to a 4.2% improvement in AUC@5° across multiple benchmarks, significantly outperforming both the strongest individual estimators and recent vision-language model-based approaches.

Abstract

A wide range of approaches have been developed for camera pose estimation, including correspondence-based methods, end-to-end pose regression, and recent 3D geometric foundation models. Our key observation is that no single estimator is optimal for diverse challenges, such as wide baselines, lack of texture, appearance changes, and occlusions. Further analysis reveals substantial performance variation across both benchmarks and individual image pairs, with different estimators exhibiting complementary strengths. We introduce PoseAgent, an agentic framework for relative camera pose estimation that dynamically orchestrates pose estimators through learnable ranking and verification. Given an image pair, a profiling agent first extracts appearance, semantic, and geometric features relevant to pose estimation, e.g., scene type. A learned ranking agent then predicts the relative competence of multiple pose estimators given the image-pair profile. The top-ranked estimator is executed, and its predicted pose is assessed by a learned verification agent that estimates the corresponding pose error. When verification fails, PoseAgent adaptively invokes lower-ranked estimators until a candidate is accepted or the execution budget is reached. For pose verification, our verification network predicts pose errors more accurately than prior models. For pose estimation, PoseAgent improves AUC@5 degree up to 4.2% over the strongest standalone estimator on each of ARKitScenes, MegaDepth, ScanNet++, and RealEstate10K. On ARKitScenes, PoseAgent also outperforms VLM-based agents, which include a VLM ranker with the same verifier and fallback policy. These results demonstrate the effectiveness of our learned ranking and verification.

Problem

• relative camera pose estimation
• pose estimator selection
• wide baseline
• occlusion
• pose verification

Innovation

• Agentic Framework
• Relative Camera Pose Estimation
• Learned Ranking
• Pose Verification
• Fallback Strategy