Senior Deep Learning Software Infrastructure Engineer

NVIDIA•Santa Clara, CA

About The Position

NVIDIA is seeking a Deep Learning Software Infrastructure Engineer to advance its Autonomous Vehicles project. The role builds and scales training libraries and infrastructure that enable end-to-end autonomous driving models. The infrastructure supports training on thousands of GPUs and massive datasets to accelerate iteration and improve safety, in collaboration with research and platform teams across NVIDIA. The engineer will develop and harden deep learning infrastructure for multi-thousand-GPU clusters; improve efficiency across data loading, distributed training, scheduling, and performance monitoring; and build robust pipelines and libraries for large video datasets and rapid experimentation. The role owns core infrastructure components, including orchestration libraries, distributed training frameworks, and fault-resilient training systems, and partners with leadership to scale infrastructure as GPU capacity and dataset sizes grow while maintaining developer efficiency and stability.

Requirements

Bachelor’s, master’s, or PhD in Computer Science, Electrical/Computer Engineering, or a related field, or equivalent experience.
12+ years of professional experience building and scaling high-performance distributed systems, ideally in ML, HPC, or large-scale data infrastructure.
Extensive knowledge of deep learning frameworks (PyTorch preferred), large-scale training (DDP/FSDP, NCCL, tensor/pipeline parallelism), and performance profiling.
Strong systems background in datacenter networking (RoCE, IB), parallel filesystems (Lustre), storage systems, and schedulers (Slurm, Kubernetes, etc.).
Proficiency in Python, including experience writing production-grade libraries, orchestration layers, and automation tools.
Ability to work closely with multifunctional teams, including ML researchers, infrastructure engineers, and product leads, and translate requirements into robust systems.

Nice To Haves

Demonstrated experience scaling large GPU training clusters with more than 1,000 GPUs.
Expertise in fault resilience and high availability, including elastic training and large-scale observability.
Hands-on technical leadership, encouraging others and establishing guidelines for ML systems engineering.

Responsibilities

Craft, scale, and harden deep learning infrastructure libraries and frameworks for training on multi-thousand-GPU clusters.
Improve efficiency throughout the training stack, including data loaders, distributed training, scheduling, and performance monitoring.
Build robust training pipelines and libraries to handle massive video datasets and enable rapid experimentation.
Collaborate with researchers, model engineers, and internal platform teams to enhance efficiency, minimize stalls, and improve training availability.
Own core infrastructure components such as orchestration libraries, distributed training frameworks, and fault-resilient training systems.
Partner with leadership to ensure infrastructure scales with growing GPU capacity and dataset size while maintaining developer efficiency and stability.