Member of Technical Staff, Evals Lead

Build AI•San Francisco, CA

20h•Onsite

About The Position

Build AI is hiring an Evals Lead to benchmark video and world-model capabilities with and without Build AI’s data. The role calls for a deep understanding of evaluation philosophy, including what evaluations can validly claim, contamination and leakage, construct validity, and whether a metric corresponds to a real capability. The successful candidate will have driven consequential evaluations that influenced what a lab trained, shipped, or collected. The Evals Lead will design rigorous benchmarks, build analytics to understand model performance and failure patterns, and assess whether dataset scaling changes capabilities and along which dimensions. The role works closely with research and dataset/quality teams so data collection and evaluation inform one another. It also involves establishing processes that improve evaluation quality, repeatability, and scale as tasks and countries are added. Build AI is a vertically integrated data company focused on Physical AI, spanning hardware, manufacturing, logistics, data collection, and model training. The team is fully in person in San Francisco’s Financial District and Shenzhen’s Nanshan district. Build AI says it has no boundaries between engineering and research and expects technical staff to contribute to both and work across disciplines as needed.

Requirements

Have shipped or driven consequential evaluations that changed training, hiring, product, or data decisions.
Understand evaluation philosophy well enough to assess and debate validity, not merely plot results.
Do not mistake a polished dashboard for an evaluation that is valid enough to inform decisions.

Nice To Haves

Experience with video, robotics, world models, or multimodal evaluations.
Designed evaluations used in a paper, product launch, or data decision.
Background in construct validity, contamination, or leakage.
Exposure to evaluation operations, including throughput, failure taxonomy, and repeatability.

Responsibilities

Design and run benchmarks for video and world models, comparing performance with Build AI data and without it under the same protocol.
Ensure comparisons are honest by using held-out tasks and checking for contamination and leakage.
Build analytics covering model performance, failure patterns, and whether scaling the dataset changes capability and on which axes.
Work with research and dataset/quality teams so data collection and evaluations inform one another.
Drive consequential evaluations that can influence plans when results change.
Design processes that improve evaluation quality, repeatability, and scalability as tasks and countries are added.

Benefits

Competitive pay.
Medical, dental, and vision packages with generous premium coverage.
$500 per month credit for waiving medical benefits.
Housing subsidy of $2,000 per month for those living within walking distance of the office.
Relocation support for those moving to San Francisco or Shenzhen.
Wellness benefits covering fitness, mental health, and more.
Daily lunch and dinner in the office.
Unlimited compute budget subject to ROI justification.
Unlimited Codex and Claude credits.
Travel.