Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response

2026-05-06

AI Summary

This work addresses the limitations of existing semantic retrieval methods for Earth observation imagery, which struggle to support natural language queries and lack globally representative paired data for contrastive training. The authors propose GeoQuery, a novel system that leverages large language models to generate text proxy descriptions aligned with the visual embedding space, thereby bridging a frozen CLAY vision encoder and user queries without requiring joint training—enabling zero-shot satellite image retrieval. Integrating prompt engineering, a two-stage retrieval pipeline (text similarity followed by visual nearest neighbors), and an Agentic Action Graphs architecture, GeoQuery achieves a 31.6% accuracy within 50 kilometers across 76 disaster site queries, rising to 50% in flood scenarios, and has been successfully deployed in the 2025 Brisbane cyclone emergency response.

Abstract

Semantic search of Earth observation archives remains challenging. Visual foundation models such as CLAY produce rich embeddings of satellite imagery but lack the natural-language grounding needed for intuitive query, and full contrastive training of a remote-sensing CLIP-style model requires paired data and compute that are unavailable at global scale. We present GeoQuery, a zero-shot retrieval system that sidesteps this constraint through prompt-aligned text proxies. Rather than training a joint encoder, we generate language descriptions for a 100k proxy subset of global Sentinel-2 tiles and optimise the description-generation prompt so that distances in the resulting text-embedding space correlate with distances in the frozen CLAY visual-embedding space. Queries are resolved in two stages, with a text-similarity search over the proxy subset followed by a visual nearest-neighbour search over worldwide CLAY embeddings. On 76 disaster-location queries covering UK floods, US wildfires, and US droughts, GeoQuery achieves 31.6% accuracy within 50 km, with the strongest performance on floods (50% within 50 km) where terrain features are well captured by RGB embeddings. Deployed within ECHO, a crisis response system using Agentic Action Graphs, GeoQuery identified vulnerable areas during Brisbane's 2025 Cyclone Alfred, with downstream flood simulations reproducing historical patterns. Prompt-aligned proxies offer a practical bridge between EO foundation models and operational retrieval when full contrastive training is out of reach.

Problem

• zero-shot retrieval
• satellite image
• natural-language query
• Earth observation
• crisis response

Innovation

• zero-shot retrieval
• prompt-aligned proxies
• joint embedding
• satellite image search
• foundation models