Software Engineer, ML Performance

NVIDIA•Santa Clara, CA

About The Position

NVIDIA is seeking a Software Engineer, ML Performance to optimize distributed training and inference workloads across the software and hardware stack. The role involves analyzing performance from models and frameworks through data loading, checkpointing, compilers, kernels, CUDA, NCCL, hosts, and networks; identifying and resolving bottlenecks or recommending solutions supported by evidence; and defining fleet-wide GPU performance metrics and the signals needed to diagnose issues. The engineer will lead investigations of performance regressions across workloads, machines, software versions, and configurations, and develop reusable analysis methods. They will evaluate automated diagnoses, identify inaccurate or incomplete findings, and build labeled cases to improve analysis quality. The role also involves working with build partners on real workloads, following findings through to improvements, and communicating product gaps to inform the roadmap. The position is based in Santa Clara, California.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
12 or more years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production.
Performance engineering experience across more than one layer of the stack, with the ability to explain the baseline, evidence, alternative explanations, and validated outcome.
Experience diagnosing performance across many jobs, machines, or tenants, rather than tuning only a single workload.
Familiarity with framework, runtime, and compiler interfaces sufficient to identify where signals should be measured.
Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and measurement effects, and determine when evidence is insufficient.
Strong programming and debugging skills, demonstrated through maintainable analysis code, tests, or repeatable investigation procedures, including when using automated development tools.
Clear communication of findings, trade-offs, and remaining uncertainty to workload owners.
Proficiency with Python and CUDA.

Nice To Haves

Experience improving both training and inference operations.
Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang, or Dynamo.
Experience moving between application, system software, driver, or hardware roles, with a full-stack perspective.
Statistical judgment applied to noisy, multi-node, fleet-scale performance data.
Experience with counterfactual analysis and workload segmentation for diagnosing regressions.
Experience building debugging or performance-analysis tools used by other engineers, with evidence that the tools improved investigation effectiveness.

Responsibilities

Analyze distributed training and inference performance across the full stack, including models and frameworks, data loading and checkpointing, compilers and kernels, CUDA and NCCL, hosts, and networks.
Identify performance bottlenecks, resolve them or recommend solutions, and provide supporting evidence.
Define performance metrics tracked across a fleet of GPUs and break them down into factors engineers can use for debugging.
Determine which signals the platform should collect first.
Drive regression analysis by segmenting workloads across machines, versions, and configurations to isolate causes, and make the analysis reusable by others.
Evaluate automated diagnoses for accuracy, misleading results, and missed problems; build labeled cases to improve analysis quality.
Work with build partners on real workloads, carry findings through to improvements, and communicate product gaps to the roadmap.
Communicate findings, trade-offs, and remaining uncertainty to workload owners.