Staff Software Engineer, ML Infra, Autonomy

Rivian•Palo Alto, CA

21h•Remote

About The Position

Rivian Autonomy is building an ML Infrastructure team to give hundreds of ML engineers a training platform they can trust at fleet scale. We are seeking a Staff Software Engineer to help design and build the platform that trains and evaluates our autonomous driving models: the control plane that schedules jobs across accelerator clusters, the storage and I/O layer that feeds them, the observability that tells us where every GPU-hour goes, and the training framework layer that lets ML engineers write model code once and run it on any silicon we operate. Autonomy at Rivian runs a training fleet of thousands of GPUs on Kubernetes over petabyte-scale sensor data, with additional accelerator types arriving in the coming months. The fleet is fully allocated, so the next step change comes from goodput — how much useful training each GPU-hour delivers. Raising it, through scheduling that keeps large gang jobs fed, a storage and I/O layer that keeps up with the accelerators, and per-workload baselines that make every optimization measurable, is the heart of this role. The control plane and training fleet already exist and serve every ML engineer in Rivian Autonomy; the scheduling, storage/I/O, observability and framework layers on top of them are largely still to be built. This is a role for someone who wants to set technical direction rather than maintain it. The platform spans four areas — job scheduling and multi-tenant cluster management, training data storage and I/O, observability and workload optimization, and the training framework layer. You will lead one or two of these areas end-to-end and contribute across the rest; we do not expect one person to be an expert in all four. You will partner closely with the model training teams who are the platform’s customers, with the Cloud Infrastructure team that owns the clusters underneath, and with the Data Infrastructure team that produces the datasets the platform serves. As an early member of the team, you will help define its technical direction, operating model and future hiring.

Requirements

5+ years of software engineering experience, or equivalent demonstrated impact, with substantial distributed-systems work in production.
Staff-level technical leadership: you identify the problems worth solving, shape strategy across teams, make pragmatic trade-offs, and drive ambiguous initiatives from evidence to production.
Hands-on experience building or operating large-scale compute or ML infrastructure — and the failure modes that only show up at scale: hung collectives, retry storms, stranded capacity, silent input-bound jobs.
Deep expertise in at least one of the following areas, with working familiarity with the others: Cluster scheduling and multi-tenancy — Kubernetes-based scheduling and resource management (Kueue, Volcano, Slurm, YARN or equivalent), including quota, fairness and preemption design. Training data storage and I/O — object-store request behavior, caching tiers, shard formats, shuffle-versus-locality trade-offs, and measuring whether a job is input-bound. Distributed training frameworks and accelerators — PyTorch distributed, Ray Train, JAX or equivalent, on GPU, TPU or Trainium: how process groups, collectives, sharding and checkpointing actually behave. Observability and performance engineering — profiling and monitoring distributed workloads from the kernel and GPU up to the scheduler, and turning measurements into optimizations.
Strong programming skills in Python and experience with at least one additional relevant language such as Go, Rust or C++.
Experience operating on a major cloud provider (AWS preferred) and on Kubernetes.
Strong communication and developer empathy, with a track record of building platforms that engineers adopt and trust; self-directed in ambiguous problem spaces.

Nice To Haves

Deep understanding of operating systems — memory management, I/O and network stack, scheduling, kernel-level debugging — as applied to debugging and optimizing distributed workloads.
Experience with large-model training and cross-GPU communication: collective communication, tensor/pipeline parallelism, NCCL performance at scale.
Experience with Ray and KubeRay internals, Kueue or Kubernetes scheduler extensions, or maintaining a patch set on top of an upstream project.
Experience with GPU profiling and performance tooling (torch profiler, Nsight Systems and Compute, DCGM, NCCL telemetry) or their counterparts on other accelerators.
Familiarity with training-native or columnar data formats (Lance, WebDataset, Parquet row groups) and with GPU-side video decode in the input path.
Experience with experiment tracking and model registry platforms (MLflow or equivalent) at scale.
Background in applying LLM agents to infrastructure operations, triage or performance tuning.

Responsibilities

Evolve our control plane into the single entry point for every accelerator cluster we operate, across cloud providers and silicon types: a user asks for N accelerators of a given type, not for a specific cluster.
Design scheduling that maximizes fleet goodput rather than queue order: gang scheduling for jobs of hundreds of nodes, multi-factor priority and fair sharing across teams, quota borrowing with enforceable reclaim, and execution-time-aware backfill.
Treat availability as an engineered system: continuous node health checking with automatic cordon, drain and replace, and automatic classification of every failed job (user, out-of-memory, communication, hardware, platform).
Design the storage tiering between object storage, shared or node-local caches and memory, and the request patterns that keep thousands of concurrent readers from overwhelming the object store.
Kill the small-file problem for good: shard formats and indexing that serve both fleet-scale shuffled random access and sequential scans, source data stored once and referenced everywhere, and the evaluation and adoption of a training-native storage format.
Make “GPU-hours lost to input wait” a first-class metric and drive it down on real jobs.
Build the monitoring and profiling toolchain at every layer a job touches — storage, node, GPU and interconnect, scheduler, and per-job metrics surfaced to the job owner — so that platform and users see the same picture.
Establish a workload taxonomy and per-type execution baselines (step time, utilization, communication fraction, input wait, checkpoint cost) that make regressions detectable and every optimization quantifiable.
Lay the groundwork for AI-assisted triage and optimization: metrics, logs and scheduler state accessible enough that an agent can be the first responder for failed jobs and propose improvements measured against the baselines.
Help build a thin, opinionated training framework layer over open-source distributed training libraries: one API for users, per-accelerator backends underneath, with golden images, validated launch recipes and a model-zoo CI that gates every release.
Run coordinated upgrade programs across the ML stack (distributed compute framework, Kubernetes operators, queueing, experiment tracking) on a cluster that is never idle.
Define the platform roadmap with the team lead — build-versus-buy decisions, boundaries with the cluster, data and model teams, and prioritization by measured impact on goodput and cost.
Work directly with ML engineers to find where the platform slows them down or fails, and turn that into improvements to the scheduler, the data path, the tooling and the documentation.
Lead architecture across organizational boundaries, communicate recommendations to engineering leadership, and mentor the engineers building and operating the platform.