MLOps Senior Lead Engineer

NVIDIA•Santa Clara, CA
•$187,500 - $442,000

About The Position

NVIDIA’s Managed AI Research Superclusters (MARS) organization deploys and operates modern GPU high-performance computing systems for NVIDIA researchers. The MLOps team is seeking a senior engineer with expertise in HPC and computational machine learning to work closely with researchers, understand their requirements, workloads, and infrastructure needs, and turn recurring problems into platform improvements. The role sets technical direction for the MLOps team while remaining hands-on in complex customer and performance situations. The engineer will diagnose and resolve end-to-end distributed training and inference problems across compute, networking, storage, containers, and schedulers; identify and address recurring customer issues at scale; improve and test the platform; and develop training and reference materials. The role also mentors engineers, supports system bring-ups, leads design reviews, and documents engineering decisions.

Requirements

  • Bachelor’s or master’s degree in Computer Science, Engineering, or a related field, or equivalent experience.
  • 8 or more years designing, operating, or supporting large-scale GPU or HPC compute infrastructure.
  • Experience with distributed training and inference as an infrastructure engineer or advanced user.
  • Strong hands-on experience with Slurm or Kubernetes for model development and prediction, and with containers such as Docker, Enroot, Apptainer, or Singularity.
  • Solid understanding of at least one distributed training framework: PyTorch, NeMo, Megatron, or JAX.
  • Experience with inference servers such as Triton, TorchServe, or Ray Serve.
  • Experience profiling distributed training performance with tools such as the PyTorch profiler or flight recorder, Nsight Systems, and Nsight Compute.
  • Proven understanding of NCCL collectives such as all-gather and all-to-all.
  • Fluency in Python and at least one systems language: Go, Rust, or C++.
  • Ability to explain how to isolate performance problems to compute, network, or storage and debug complex systems.
  • Data-driven judgment and ability to influence without authority across research, platform, and partner teams.
  • Excellent written and verbal communication.

Nice To Haves

  • Familiarity with infrastructure as code such as Terraform.
  • Experience leading bring-up or acceptance testing of GPU clusters with thousands of nodes, including MLPerf or similar benchmarking.
  • Deep knowledge of InfiniBand and RDMA, and fast distributed storage such as Lustre or GPFS for AI and HPC workloads.
  • CUDA programming background and understanding of how GPUs, networks, and storage interact during training.
  • Experience with observability for ML workloads, including OpenTelemetry, Grafana, or ClickHouse, and with goodput or efficiency metrics.
  • Track record of technical leadership, including mentoring engineers, shaping team engineering practices, and engaging partners and vendors.

Responsibilities

  • Set the technical direction for the MLOps team and keep the team focused on improving customer efficiency.
  • Serve as a senior hands-on engineer in complex customer and performance situations.
  • Work directly with researchers on LLM, autonomous-vehicle, and generative-model workloads to diagnose training and inference performance issues and recurring crashes across compute, network, and storage.
  • Debug large-scale distributed training end to end, including NCCL collectives, high-speed interconnects, RDMA, parallel storage, containers, and Slurm and Kubernetes schedulers.
  • Identify recurring problems across customers and work with MARS peers, partners, and platform teams to resolve them at scale.
  • Translate customer needs into platform feature requests.
  • Test the platform using customer workloads and move testing earlier so issues are caught before capacity is handed over.
  • Produce training, guidance, and reference tests; share findings through blog posts, talks, and papers.
  • Mentor engineers at all levels, pair on bring-ups, run design reviews, and ensure decisions are documented with a clear owner.

Benefits

  • Equity
  • Comprehensive benefits package
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service