Senior Software Engineer, ML Evaluation & Introspection Infrastructure

Aurora•Mountain View, CA

1d•$162,000 - $260,000•Hybrid

About The Position

Aurora is seeking a Senior Software Engineer to join its Behavior Analysis team and report to the team’s engineering manager. The team develops evaluation, embedding-search, and introspection systems that help determine whether each new version of the learned model powering Aurora’s self-driving trucks is better and safer than the previous version. Engineers use the team’s tools to build more than 100 training and evaluation datasets each week. The team is small, with engineers owning complete systems, and designs its tools so both engineers and AI agents can use them effectively. The role will own core parts of the model analysis and evaluation loop, beginning with embedding generation and search. It will also help advance the use of LLMs, VLMs, and agents to analyze driving scenarios and evaluation results, with the aim of making analyses that currently take days possible in minutes. The position involves close collaboration with ML engineers to decide what to measure and to make those measurements trustworthy, observable, and economical to run on candidate models. For roles based in Mountain View, California, the salary range is $162,000–$260,000 per year. The successful candidate is also eligible for an annual bonus, equity compensation, and benefits. Aurora expects employees to work from the office at least three days per week; some roles may require up to five days per week based on operational needs.

Requirements

Typically 5 or more years of professional software engineering experience building backend, data, or ML infrastructure.
Strong Python skills and proficiency with SQL.
Experience designing and operating large-scale batch or data pipelines on cloud infrastructure, including backfills, retries, and cost management.
Experience building systems that measure or evaluate something, such as metrics, experimentation, evaluation, or analytics platforms.
Experience with distributed computing and container orchestration, such as Kubernetes, AWS Batch, or similar.

Nice To Haves

Experience with ML evaluation infrastructure, such as evaluation harnesses, golden datasets, regression detection, or A/B model comparison.
Experience with embeddings and vector search, or model introspection and interpretability tooling.
Experience using LLMs, VLMs, or agentic workflows to automate analysis, labeling, or data curation.
Experience in autonomous vehicles, robotics, or other large-scale sensor data.
Experience with observability for data systems, including metrics, alerting, and SLOs, and everyday use of AI coding agents.

Responsibilities

Build and scale evaluation pipelines that compare model versions on large driving datasets, including offline evaluations, slice-based metrics, regression detection, and model-versus-model reports.
Own embedding and vector-search pipelines over driving data, versioned per model, so engineers can search, cluster, and compare model representations.
Build model introspection infrastructure, including attention, attribution, and ablations, to make model behavior queryable.
Help define how LLMs, VLMs, and agents can support ML analysis, including triaging, labeling, clustering, and summarizing driving scenarios and evaluation results.
Collaborate with ML engineers to determine what to measure and make measurements trustworthy, observable, and economical to run on every candidate model.
Build the web application and CLI engineers and AI agents use to find datasets, training runs, evaluations, and their costs.