Software Engineer, Applied AI- Evals

Mercor•San Francisco, CA

1d•$220,000 - $250,000•Onsite

About The Position

Mercor’s evals platform helps frontier labs and enterprises understand how models perform on real work. The platform runs evaluations at scale, grades results, and explains model failures. The team also operates APEX, a family of benchmarks created with domain experts to measure frontier models’ performance on professional work in fields such as law, finance, and consulting. A public leaderboard ranks models, and APEX results appear in model cards across the industry. As a Software Engineer on the Evals product team, you will build scorecards, failure reports, and dataset views that help customers turn evaluation runs into decisions. You will address challenges involving scale, grading accuracy, and reproducibility while keeping the product easy to use. The role may expand from evaluations into post-training and other product areas, helping turn identified failures into data that improves models.

Requirements

Bachelor’s degree in Computer Science or a related field.
Strong full-stack skills in a modern backend language such as Python, Go, or TypeScript, and a frontend framework such as React.
Track record of owning product features end to end, from idea to production.
Strong product sense focused on user needs.
Experience building data-heavy products, including data models, APIs, and views over large volumes of results.
Excellent written and verbal communication with engineers, researchers, and customers.
Comfort with ambiguity and a habit of shipping iteratively.

Nice To Haves

Experience with LLM evaluations, benchmarks, or AI developer tools.
Startup experience or experience building B2B developer products.

Responsibilities

Build APIs, data models, and UI that help customers run evaluations, compare models, and share scores.
Run every new frontier model on APEX and publish accurate, defensible results to the public leaderboard.
Build failure analysis that shows customers where models fail and links each pattern to the exact steps that went wrong.
Build dataset views, registries, and versioning to help customers find benchmarks or bring their own.
Build onboarding, roles, key management, and usage views so new teams can run evaluations independently.
Join customer pilots with labs and enterprises and translate their needs into the product roadmap.