AI Performance Engineer -GPU infrastructure
About The Position
The AI Performance Engineer will lead performance benchmarking, analysis, and tuning of provisioned AI clusters across distributed training, inference, and GPU collective communication. This senior specialist role establishes reproducible baselines, identifies performance gaps against comparable reference configurations, and collaborates with platform and network teams to resolve bottlenecks. The engineer validates improvements while preserving model quality, numerical correctness, and production stability. The role delivers automated benchmark suites, reproducible baseline reports, profiling evidence, prioritized remediation plans, validated tuning configurations, and performance regression tests. Success is measured by documented closure of gaps against comparable references, improved throughput and scaling or reduced latency, and repeatable results that preserve agreed model quality and numerical correctness.
Requirements
- At least 7 years in performance engineering, HPC, ML systems, or GPU infrastructure, including hands-on benchmarking of distributed training and inference workloads.
- Strong Python and Linux skills for benchmark automation, evidence collection, system diagnostics, and repeatable test execution.
- Practical C++/CUDA understanding.
- Experience with PyTorch, GPU profiling, collective communication, and distributed execution, including the impacts of parallelism, synchronization, memory use, and data movement.
- Hands-on use of training and inference benchmarking tools, with ability to select representative workloads and interpret throughput, latency distributions, and scaling behavior.
- Knowledge of benchmark statistics and experimental controls, including warm-up, repeated measurements, run-to-run variation, and fair comparison against reference results.
- Ability to isolate cross-layer bottlenecks, validate performance changes without compromising correctness, and explain findings and remediation priorities to engineering stakeholders.
Nice To Haves
- Experience qualifying NVIDIA AI clusters at scale, including GPU topology, InfiniBand/RoCE, RDMA, NCCL behavior, and hardware, driver, and runtime compatibility.
- Experience with NeMo/Megatron training, MFU analysis, TensorRT-LLM, vLLM/SGLang, NVIDIA Dynamo, and AIPerf or GenAI-Perf.
- Experience integrating benchmark suites with deployment and release validation, using Nsight and DCGM evidence to close performance gaps and detect regressions.
- Preferred credentials: NCP-AIO, NCP-GENL specialization, and NCA-AIIO foundational certification. Candidates should demonstrate corresponding skills through practical benchmarking, profiling, and performance-tuning experience.
Responsibilities
- Define benchmark plans and acceptance criteria for single-node and multi-node workloads.
- Compare benchmark results with references using equivalent hardware, models, precision, software versions, and workload settings.
- Build automated benchmark suites using MLPerf-style methodologies, NCCL Tests, NeMo/Megatron, TensorRT-LLM/trtllm-bench, NVIDIA AIPerf, and vLLM/SGLang load testing.
- Measure training throughput, step time, model FLOPs utilization (MFU), scaling efficiency, and collective bandwidth and latency.
- Track inference throughput, time to first token, inter-token and tail latency, and GPU memory use.
- Profile bottlenecks with PyTorch Profiler, Linux perf, NVIDIA Nsight Systems/Compute, and DCGM.
- Correlate workload behavior with compute, memory, storage, network, and host-level evidence.
- Tune batch sizes, sequence lengths, data loading, tensor/data/pipeline parallelism, and communication overlap.
- Optimize serving concurrency, batching, and KV-cache settings for representative traffic.
- Evaluate supported precision and quantization options against model quality and numerical correctness requirements.
- Document throughput, latency, memory, and quality tradeoffs before recommending changes.
- Investigate NCCL, GPU interconnect, RDMA, topology, and host configuration issues with platform and network teams.
- Distinguish application and framework bottlenecks from infrastructure limitations.
- Control warm-up and repeated runs, report variability, and check for load-generator bottlenecks.
- Deliver profiling evidence, prioritized remediation, validated configurations, and automated performance regression tests.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
