Senior Software Engineer, ML Evaluation & Introspection Infrastructure

Aurora•Seattle, WA

2d•Hybrid

About The Position

Aurora is seeking a Senior Software Engineer to join its Behavior Analysis team and report to the team’s engineering manager. The team builds systems used to assess and compare learned driving models on real-world driving data, including evaluation tools, embedding search, and model-introspection systems. Engineers use these tools to build more than 100 training and evaluation datasets each week. The team is small, with engineers owning end-to-end systems, and designs its tools for both engineers and AI agents. The engineer will own core parts of this evaluation and analysis infrastructure, beginning with embedding generation and search. The role also contributes to expanding the use of LLMs, VLMs, and agents to analyze driving scenarios and evaluation results, helping automate work such as triaging, labeling, clustering, and summarization. The position involves working closely with ML engineers to define measurements and make them trustworthy, observable, and economical to run for candidate models. Aurora expects employees to work from the office at least three days per week. Some roles may require up to five days in office based on operational needs. The position is offered in Mountain View, California; Seattle, Washington; and Pittsburgh, Pennsylvania. Compensation includes a location-dependent base salary range, annual bonus eligibility, equity compensation, and benefits.

Requirements

Typically 5 or more years of professional software engineering experience building backend, data, or ML infrastructure.
Strong Python skills and proficiency with SQL.
Experience designing and operating large-scale batch or data pipelines on cloud infrastructure, including backfills, retries, and cost management.
Experience building systems that measure or evaluate something, such as metrics, experimentation, evaluation, or analytics platforms.
Experience with distributed compute and container orchestration, such as Kubernetes, AWS Batch, or similar.

Nice To Haves

Experience with ML evaluation infrastructure, such as evaluation harnesses, golden datasets, regression detection, or A/B model comparison.
Experience with embeddings and vector search, or model-introspection and interpretability tooling.
Experience using LLMs, VLMs, or agentic workflows to automate analysis, labeling, or data curation.
Experience with autonomous vehicles, robotics, or other large-scale sensor data.
Experience with observability for data systems, including metrics, alerting, and SLOs.
Everyday use of AI coding agents.

Responsibilities

Build and scale evaluation pipelines that compare model versions on large driving datasets, including offline evaluations, slice-based metrics, regression detection, and model-comparison reports.
Own embedding and vector-search pipelines over driving data, versioned per model, enabling engineers to search, cluster, and compare model representations.
Build model-introspection infrastructure, including attention, attribution, and ablations, to make model behavior queryable.
Help define and implement uses of frontier LLMs, VLMs, and agents for analyzing, triaging, labeling, clustering, and summarizing driving scenarios and evaluation results.
Work with ML engineers to determine what to measure and make measurements trustworthy, observable, and economical to run on each candidate model.
Build the web application and CLI that engineers and AI agents use to find datasets, training runs, evaluations, and associated costs.