Principal Software Engineer, E2E Performance and Goodput — CSP Engagements

NVIDIA•Austin, TX

1d•$272,000 - $431,250

About The Position

NVIDIA is seeking a Principal Engineer for its CSP Engagements team to serve as the technical focal point for end-to-end performance. The role works directly with engineering teams at key cloud service provider (CSP) and hyperscale customers to help them meet performance targets on NVIDIA platforms. It augments NVIDIA's performance and benchmark teams with a dedicated customer-facing focus, building shared understanding of platform performance characteristics, incorporating workload-specific customer feedback into NVIDIA's optimization priorities, and validating performance in customer-representative configurations. Cross-CSP visibility is used to identify patterns and drive improvements in documentation, configuration guidance, and tooling. The engineer will drive performance characterization and validation work, collaborate with CSPs on profiling and benchmarking, and relay optimization priorities to NVIDIA engineering teams. The position also ensures performance tools and integration work are ready for customer deployments and defines test strategies for internal certification and customer acceptance.

Requirements

15+ years of experience in systems performance engineering, ideally in GPU, HPC, or ML infrastructure.
Bachelor's or master's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
Proficiency in GPU workload profiling using Nsight Systems, Nsight Compute, DCGM metrics, or equivalent instrumentation.
Understanding of distributed training performance dynamics, including computation/communication overlap, pipeline bubbles, memory bandwidth utilization, and collective efficiency.
Knowledge of statistical methods for performance analysis, including regression detection, confidence intervals, and large-scale A/B comparison.
Understanding of how the full software stack affects performance, including driver overhead, collective algorithm selection, memory allocation, scheduling, and firmware power management.
Strong data analysis and visualization skills, including Python, pandas, and dashboards.
Customer-focused approach and genuine interest in understanding and resolving why customers are not achieving expected performance.
Ability to communicate performance findings to both deeply technical audiences and executive leadership.
Demonstrated success influencing multiple engineering teams to prioritize performance improvements.

Nice To Haves

Experience profiling and optimizing distributed training at 1,000+ GPU scale, including Megatron-LM, DeepSpeed, or FSDP.
Background in ML infrastructure performance at a CSP or hyperscaler.
Familiarity with NVIDIA platforms such as DGX, HGX, and NVLink topology, and with profiling tools.
Experience building automated performance regression detection systems for production environments.
Understanding of inference workload performance dynamics, including vLLM, TensorRT-LLM, SGLang, and continuous batching.

Responsibilities

Drive performance characterization work streams with engineering teams at key CSP and hyperscale customers, helping them understand performance expectations, profiling methodology, and workload-specific tuning options.
Gather and synthesize CSP performance feedback, identify gaps between expected and actual throughput, and champion optimization priorities to NVIDIA's CUDA, NCCL, driver, and firmware teams.
Ensure key open-source performance and stress tools, including STREAM, GPU Burn, and GPU BLAST, are updated and validated for the latest NVIDIA rack-scale systems, GPU architectures, and CPU platforms.
Work with CSPs to ensure their performance and validation tooling reflects the latest GPU capabilities, memory hierarchy changes, and platform-specific tuning parameters.
Conduct cross-CSP performance comparisons and pattern analysis to identify configuration, software, or workload differences that explain performance gaps.
Collaborate with CSPs to ensure performance-related integration work, including profiling infrastructure, benchmark harnesses, and configuration validation, is ready ahead of deployment milestones.
Define test strategies and tooling requirements for performance validation for both NVIDIA internal certification and customer acceptance.