Tech Lead, Applied AI - Evals

Mercor•San Francisco, CA

20h•$260,000 - $500,000•Onsite

About The Position

Mercor is hiring a Tech Lead for its Evals team to own the evals product end to end, from technical direction and roadmap through hands-on implementation. Mercor's evals product helps frontier labs and enterprises understand model performance on real work. It runs evaluations at scale, grades results, and explains model failures. The team also runs APEX, a family of benchmarks built with domain experts to measure frontier models' performance on professional work, with results published on a public leaderboard and appearing in model cards across the industry. The role focuses on reliably grading open-ended professional work, maintaining reproducible results as models and tasks change, and running long agent tasks at scale. The Tech Lead will set architecture and priorities, build product capabilities used by labs and enterprises, and ensure evaluation results are correct and defensible. The position will expand with the platform into post-training and new product areas, helping turn identified model failures into data that improves models. Mercor describes itself as a profitable Series C company valued at $10 billion. The team works in person five days a week in its San Francisco, New York City, or London offices.

Requirements

Bachelor's degree in Computer Science or a related field.
At least 5 years building and running backend systems in production, including time at senior or staff level.
Strong full-stack skills in a modern backend language such as Python, Go, or TypeScript, and a frontend framework such as React.
Track record of owning a platform or product area end to end, including leading the team that built it.
Strong product sense grounded in user needs.
Experience building data-heavy products, including data models, APIs, and views over large volumes of results.
Excellent written and verbal communication with engineers, researchers, and customers.
Comfort with ambiguity and a habit of shipping iteratively.

Responsibilities

Own the evals flywheel and its direction by setting architecture and roadmap and making tradeoff decisions in design reviews.
Lead the product customers use to evaluate and compare models, including shipping APIs, data models, and UI.
Build systems to run every new frontier model on APEX and publish correct, defensible results to the public leaderboard.
Build failure analysis that links each failure pattern to the exact steps that went wrong.
Set the quality bar for evaluations by calibrating graders against expert judgment and deciding when grader or agent changes are ready to ship.
Maintain product reliability and keep compute costs in check.
Lead pilots with labs and enterprises and incorporate customer learnings into the roadmap.
Hire and mentor engineers and manage the team.