Senior Engineer: Building internet scale Web crawls

Engineering: Internet-Scale Web Crawling & Data Infrastructure

Location: Remote / Flexible

Type: Full-time

Salary: Highly competitive: up to $250K DOE + Equity

Our client is building next-generation AI systems that rely on massive, high-quality web data. We’re looking for engineers who have designed, built, or operated large-scale web crawlers. Systems that operate reliably across millions or billions of URLs.

This is a hands-on role for people who have worked close to the metal on web-scale data acquisition, not just consumed third-party datasets.

Well funded exciting start up firm who are leading the way in Web Data and AI!

Experience Required:

1. Designing and building internet-scale web crawling pipelines
2. Handling real-world challenges like:
3. Rate-limiting, bot detection, CAPTCHAs, and dynamic content
4. Distributed crawling, scheduling, and prioritisation
5. Fault tolerance and crawl resumption at massive scale
6. Extracting, normalising, and validating large volumes of web data
7. Optimising for performance, cost, and data quality
8. Working closely with AI and research teams to support model training and evaluation

Must have:

1. Built or operated large-scale crawlers (millions+ of pages)
2. Worked on search engines, data platforms, ad tech, AI datasets, or web archiving
3. Designed distributed systems for high-throughput data ingestion
4. Dealt with hostile or semi-hostile web environments

Strong experience with:

1. Python, Go, Rust, or Java
2. Distributed systems (queues, workers, schedulers)
3. HTTP, HTML parsing, JS rendering (Playwright, Puppeteer, Selenium, etc.)
4. Cloud infrastructure (AWS, GCP, or similar)

Package and benefits

1. Work on hard, unsolved problems at true internet scale
2. Direct impact on cutting-edge AI systems
3. High-calibre engineering team
4. Competitive compensation and meaningful equity

If you are experience and passionate about building internet scale web crawls, APPLY ASAP!

Are you looking for remote jobs near your area? At Yulys, thousands of employers are looking for exceptional talent like yours. Find a perfect job now.