ML Evals Engineer

Exa • San Francisco, CA

71d

About The Position

Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in rust to search over it. We also own a $5m H200 GPU cluster and routinely run batch jobs with 10s of thousands of machines. This isn't your average startup. On the ML team, we train foundational models for search. Our goal is to build systems that can instantly filter the world's knowledge to exactly what you want, no matter how complex your query. Basically, put the web into an extremely powerful database. We're looking for an ML evals engineer to design and build our eval stack at Exa. The role involves investigating how to evaluate search engines in an LLM world and then building the most comprehensive, creative, and effective eval suite. You will be deciding the future of search through the evals we choose to optimize for.

Requirements

• Some experience in machine learning (ML).
• Strong engineering experience.
• Experience in creating evaluation datasets and analyzing data.
• A deep interest in the problem of search and a desire to improve search engine performance.

Nice To Haves

• Experience with large-scale infrastructure and batch jobs.
• Familiarity with vector databases and embedding models.

Responsibilities

• Design and build the evaluation stack for search engines.
• Investigate methods to evaluate search engines in the context of LLMs.
• Create comprehensive and effective evaluation suites for search.
• Identify the biggest problems in search and develop evaluations for those issues.
• Think creatively about gathering evaluation data.

Benefits

• Sponsorship for international candidates (e.g., STEM OPT, OPT, H1B, O1, E3).

What This Job Offers

• Career Level: Entry Level