Software Engineer, Performance Instrumentation

NVIDIA•Santa Clara, CA

About The Position

NVIDIA is seeking a Software Engineer, Performance Instrumentation to join its team in Santa Clara, California. The role supports work at the intersection of AI and accelerated computing, with a focus on the performance data and instrumentation needed to diagnose and analyze systems and machine-learning workloads. The engineer will define performance data requirements across profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry. They will identify and demonstrate visibility gaps, work with partner teams to address them, and guide the development of low-overhead instrumentation and annotation APIs. The role also ensures the quality of ingested data, measures instrumentation overhead and observer effects on real workloads, and coordinates delivery across multiple partner teams. The position requires a bachelor’s degree in a relevant field or equivalent experience, at least eight years of relevant software engineering experience, and hands-on instrumentation or tracing experience in production machine learning or systems software. The base salary range is $168,000–$270,250, and the role is also eligible for equity and benefits.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
At least 8 years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
Proficiency in CUDA and working knowledge of machine learning frameworks such as PyTorch or JAX, and annotation or tracing APIs such as NVTX.
Experience maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
Ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
Demonstrated ability to implement modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and following through to delivery.
Strong debugging skills across component boundaries and ability to validate changes through code review, tests, and independent measurements, including with automated development tools.
Excellent communication skills, particularly in conveying evidence, uncertainty, and next steps to engineers and data users.
Proficiency in Python and C++.

Nice To Haves

Experience establishing telemetry conventions or schemas adopted by other teams.
Background in storage, checkpointing, or data-loading performance.
Expertise in crafting low-overhead event collection.
Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.

Responsibilities

Define and maintain performance data requirements across profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
Identify performance visibility gaps and provide concrete evidence of diagnoses that cannot be made with current data.
Collaborate with relevant teams to close visibility gaps or develop solutions where needed.
Inform the development of low-overhead instrumentation and annotation APIs, ensuring they efficiently convey information needed by consumers.
Implement ingestion validation, sanitization, checks, and alerting to detect data gaps or errors before they affect users or automated analysis.
Measure and manage instrumentation overhead and observer effects on real workloads.
Coordinate with multiple partner teams, manage commitments, and ensure continuous progress.
Debug across component boundaries and validate changes through code review, tests, and independent measurements.