Software Engineer, Web Crawling
Exa•San Francisco, CA
•$180,000 - $350,000•Onsite
About The Position
Exa is an applied AI lab building a search engine and large-scale infrastructure to crawl the web, process it with embedding models, and retrieve information using vector databases. The company says its search technology powers Cursor, Cognition, HubSpot, and more than 400,000 developers. The role is an in-person opportunity in Zurich, focused on building and operating web-crawling systems at Google scale. Exa is willing to sponsor international candidates, though visa sponsorship is not guaranteed.
Requirements
- Extensive experience building and scaling web crawlers, or the ability and willingness to ramp up very quickly.
- Experience with a high-performance language such as C++ or Rust.
- Familiarity with TypeScript, Playwright, modern web design, and the Chrome DevTools Protocol (CDP).
- Comfort optimizing a system to an exceptional degree.
- Care about finding high-quality knowledge and recognize its importance.
Responsibilities
- Build a distributed crawler capable of handling more than 100 million pages per day.
- Optimize crawl politeness and rate limiting across thousands of domains.
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures.
- Create intelligent crawl scheduling and prioritization algorithms to maximize coverage efficiency.
Benefits
- Premium medical, dental, and vision healthcare benefits.
- Fertility benefits.
- 16 weeks of fully paid parental leave for all new parents.
- Monthly wellness stipend.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Education Level
No Education Listed
