About The Position

Runway is seeking an ML infrastructure engineer to own model evaluation end to end and build a unified evaluation platform. The role will develop systems to generate model samples at scale, score them using automated metrics and human annotations, track results across checkpoints and releases, and deliver results to researchers quickly. The engineer will define operational standards for measuring model quality, assessing confidence, and determining when results support shipping decisions. Embedded within research teams as part of ML Platform, this role will set technical direction for evaluations across the company. It will work with researchers across video, image, audio, agents, and robotics, translating their measurement needs into reusable platform capabilities. The platform will support research and integrate with training, production model serving, and other company efforts. Runway’s inference and training code uses Python and PyTorch in a cloud-native environment, with multiple clouds and Kubernetes-native and custom job orchestration tooling. The role has access to models, agents, GPUs, storage, and cloud services for building the evaluation pipeline.

Requirements

  • At least 5 years of experience building ML infrastructure or data platforms in production environments, including some experience with evaluation, experimentation, or benchmarking systems.
  • Strong Python and PyTorch skills.
  • Hands-on experience running large batch GPU workloads on Kubernetes.
  • Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility.
  • Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple-comparison pitfalls, and inter-rater agreement.
  • Comfort building internal tools end to end, from the command line to the browser.
  • Ability to lead a broad technical area, gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap independently.
  • Familiarity with the full model development lifecycle, including data, training, evaluation, and serving.
  • Ability to work as a self-starter, embed with research teams, and move fast.
  • Strong systems thinking and a pragmatic approach to production reliability.
  • Humility and open mindedness, with a willingness to learn from others.

Nice To Haves

  • Track record of building evaluation suites for generative models.
  • Hands-on experience with LLM- or VLM-as-judge pipelines.
  • Experience with online experimentation platforms and connecting offline metrics to product outcomes.
  • Prior experience evaluating agents or robotics policies.

Responsibilities

  • Own the evaluation platform end to end, including systems for generating, annotating, reviewing, and adapting evaluations at frontier scale.
  • Define the CLIs, APIs, GUIs, and storage layers needed to make the evaluation process seamless, fast, sophisticated, and collaborative.
  • Work directly with research teams in video, image, audio, agents, and robotics to understand evaluation needs, build solutions, and generalize them into the platform.
  • Set standards for model evaluation, including reproducibility, metric definitions, reporting formats, and determining when results are trustworthy enough to act on.
  • Support broad adoption of the platform and its integration across research efforts, training, production model serving, and other areas.
  • Contribute to ML Platform tools and systems that help Runway train and serve frontier models.
  • Gather requirements from teams, set technical direction, make tradeoffs, and drive an evaluation roadmap.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service