Software Engineer, ML Training Performance

Zoox•Foster City, CA

1d•$177,000 - $270,000

About The Position

Zoox’s ML Platform team builds and operates the foundational tools, model-development systems, and training infrastructure used by applied research teams for in-vehicle and off-vehicle ML applications. The team supports work on large-scale foundation models, vision-language models (VLMs), and vision-language-action models (VLAs) for autonomous driving. In this role, you will advance ML training performance by applying distributed-training strategies, mixed precision, gradient checkpointing, reinforcement learning (RL) optimization, and GPU-kernel techniques. You will profile and debug throughput bottlenecks across compute, compilers, frameworks, and cluster networking, and work with researchers, data engineers, and hardware engineers on requirements and architecture. You will also implement custom kernels and benchmarks to measure and validate performance improvements. The role partners with Zoox Autonomy teams, including Perception, Prediction, Planner, Simulation, and Collision Avoidance.

Requirements

At least 4 years of total experience, including at least 2 years working on large-scale model-training platforms.
Experience with distributed-training frameworks such as Megatron-LM or TorchTitan and efficiently using GPU or accelerator clusters at scale.
Experience with reinforcement-learning frameworks such as Verl or MILES and RL algorithms such as GRPO or PPO.
Experience with collective communication libraries such as NCCL and optimizing communication/compute overlap in multi-node training.
Experience with profiling tools such as NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling to identify and resolve training bottlenecks.
Proficiency in Python or C++.

Nice To Haves

Experience with ML compilers, performance tooling, or hardware/software co-design, such as GPU, TPU, or accelerator architectures.

Responsibilities

Design, implement, and operate ML training performance optimizations, including data, tensor, and pipeline parallelism, mixed-precision training, gradient checkpointing, and GPU kernels.
Optimize training efficiency for VLM, VLA, and foundation models.
Work on reinforcement-learning frameworks and performance optimization for driving foundation models.
Profile and debug training-throughput bottlenecks across compute, compiler, framework, and cluster-networking layers.
Collaborate with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.
Partner with Zoox Autonomy teams, including Perception, Prediction, Planner, Simulation, and Collision Avoidance.