Senior Software Engineer, Autonomy Evaluation & Metrics Platform

Rivian • Palo Alto, PA

7h

About The Position

Rivian’s Autonomy Eval Infrastructure team builds software that helps engineers understand and improve the ML models behind its driving systems and provides release teams with trusted evidence. Within Autonomy’s AI Platform organization, the team connects model development with evaluation workflows, metrics, triage, and visualization across simulation and real-world vehicle data.

The Senior Software Engineer will design, build, and operate a substantial part of the autonomy evaluation platform, making evaluations consistent and reproducible. Working directly with Perception, Planning, and Systems engineers, this role connects model experiments to reliable metrics and uses insights about model behavior, failure modes, and generalization to guide improvements to models and data.

This is a hands-on software engineering role with ownership of projects from design through implementation, rollout, and operation. The engineer will make practical system-design trade-offs, mentor teammates, and help establish engineering practices and interfaces as an early member of a newly formed team.

Requirements

• At least 5 years of professional software engineering experience, or equivalent demonstrated experience and impact, including ownership of production backend or data-intensive systems.
• Hands-on experience building software for autonomy or robotics, with direct exposure to evaluation, simulation, perception, planning, or validation workflows.
• Strong Python programming skills, including writing, testing, profiling, and debugging maintainable production software.
• Practical SQL skills, including validating query logic, results, and performance; AI coding tools are expected to support writing and refining queries.
• Demonstrated system-design ability across service APIs, data modeling, distributed processing, and storage, including failure recovery, consistency, and performance trade-offs.
• Experience taking a project from requirements and design through implementation, rollout, and operation; coordinating dependencies and communicating trade-offs clearly.
• Practical understanding of model inference and evaluation, including held-out datasets, generalization, reproducibility, model versioning, and differences between open-loop replay, closed-loop simulation, and real-world measurements.
• Ability to reason about sampling, confidence intervals, noisy metrics, and valid comparisons, and to work with domain experts when stronger statistical analysis is needed.
• Experience with cloud infrastructure, containers, CI/CD, and production observability, plus a track record of constructive code review and mentoring.

Nice To Haves

• Experience building metric registries, evaluation services, experiment-tracking systems, or automated regression gates used by multiple engineering teams.
• Experience with Spark, Databricks, Delta Lake, or comparable distributed data systems, and with AWS, Docker, or GitLab CI.
• Experience migrating analytical or reporting systems without breaking downstream consumers, or linking metrics to scenario-level debugging and triage.
• Experience with PyTorch or similar ML frameworks, evaluation of end-to-end driving models, or connecting offline model metrics to closed-loop and real-world driving behavior.

Responsibilities

• Own the design and delivery of a metrics-platform area, such as the metric registry, computation services, or comparison and reporting APIs, in partnership with the team’s technical lead.
• Design versioned metric definitions and data models for inputs, aggregation, slicing, and results; preserve provenance connecting official evaluation runs to code, model, dataset, and scenario versions.
• Build distributed computation and storage paths with idempotent execution, incremental processing, and controlled backfills, balancing throughput, latency, cost, and reproducibility.
• Build comparison tools that expose coverage, sample sizes, uncertainty, and changes in evaluation inputs, helping users distinguish model regressions from data, simulator, or infrastructure failures.
• Partner with model developers to design evaluation experiments, inspect predictions, and analyze failure patterns; use findings to guide model and data improvements and build reusable evaluation harnesses.
• Integrate metrics and regression checks into replay and closed-loop evaluation workflows and CI; partner with domain owners on metric semantics and thresholds and with Systems on evidence for release decisions.
• Deliver self-service APIs and scorecards backed by consistent definitions and results; incrementally migrate existing consumers while preserving reporting continuity and traceability.
• Own service reliability and performance through testing, observability, incident investigation, and operational documentation; measure improvements in turnaround time, correctness, and cost.
• Write and review production code, lead design reviews for projects, mentor engineers, and collaborate across Palo Alto, London, and Belgrade to deliver software users adopt.

Benefits

• Medical and prescription drug insurance for full-time employees and eligible family members.
• Dental insurance for full-time employees and eligible family members.
• Vision insurance for full-time employees and eligible family members.
• Coverage effective on the first day of employment; Rivian covers most of the premiums.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed