Staff / Senior Software Engineer (Agentic Search) - Crawler

We are looking for a Senior Software Engineer to work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption.

In this role, you will focus on building systems that discover, fetch, and continuously refresh content from the open web and other large-scale data sources. You will design distributed crawling, scheduling, and ingestion infrastructure capable of operating at internet scale while balancing coverage, freshness, resource efficiency, and reliability. You will work on systems that process billions of URLs, manage high-throughput data flows, and ensure that high-quality content is consistently available to downstream indexing and retrieval systems.

Design, implement, and operate web-scale crawling systems for acquiring content from the internet

Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations

Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies

Build systems for URL discovery, deduplication, content extraction, and crawl orchestration

Ensure reliable operation of crawling infrastructure under high-throughput conditions

Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality

Monitor resource usage, bandwidth consumption, and infrastructure cost

Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements

Enable safe experimentation with crawling strategies and content acquisition policies