Senior/Staff Software Engineer, ML Training Performance
About The Position
Zoox’s ML Platform team enables innovation in large-scale Foundation models, VLMs, and VLAs for autonomous driving. In this role, you will own ML training performance and help make models train faster, more cost-effectively, and at greater scale. You will set technical direction for distributed training, parallelism strategies, mixed-precision training, and reinforcement learning on state-of-the-art accelerators, with a focus on multimodal VLA and VLM models. You will partner with Autonomy teams including Perception, Prediction, Planner, Simulation, and Collision Avoidance, and help shape the team’s roadmap as it expands into new ML domains. You will lead the design, implementation, and operation of training-performance optimizations; own the technical strategy and multi-quarter roadmap for reinforcement-learning frameworks and performance; profile and diagnose throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack; and collaborate with research, data, and hardware teams on requirements and architectural decisions. You will also build custom kernels and benchmarks, mentor engineers, and promote strong engineering practices.
Requirements
- 7+ years of total experience, including 4+ years working on large-scale model-training platforms, with a record of leading projects end to end.
- Deep experience with distributed-training frameworks such as Megatron-LM or TorchTitan and efficiently using GPU or accelerator clusters at scale.
- Experience with reinforcement-learning frameworks such as Verl or MILES and algorithms such as GRPO or PPO.
- Experience with collective-communication libraries such as NCCL and optimizing communication/compute overlap in multi-node training.
- Strong command of profiling tools such as NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling to identify and resolve training bottlenecks.
- Proficiency in Python or C++.
- Ability to influence across teams and communicate technical tradeoffs to researchers and engineers.
Nice To Haves
- Experience with ML compilers, performance tooling, or hardware/software co-design involving GPU, TPU, or accelerator architectures.
Responsibilities
- Lead the design, implementation, and operation of training-performance optimizations, including data, tensor, and pipeline parallelism, mixed precision, gradient checkpointing, and GPU kernels, to scale VLM, VLA, and Foundation model training.
- Own the technical strategy and multi-quarter roadmap for reinforcement-learning frameworks and performance optimization for driving foundation models.
- Profile and perform root-cause analysis of throughput bottlenecks across compute, compiler, framework, and cluster-networking systems.
- Partner with ML researchers, data engineers, and hardware engineers to define requirements and make architectural decisions for large-scale training infrastructure.
- Build custom kernels and benchmarks to track and validate performance gains across the stack.
- Mentor engineers and raise the team’s technical bar through design reviews and best practices.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
