Software Engineer, Performance Instrumentation

NVIDIA•Austin, TX

1d•$168,000 - $270,250

About The Position

NVIDIA is hiring a Software Engineer, Performance Instrumentation to join its team in Santa Clara, California. The role contributes to projects at the intersection of AI and accelerated computing, with a focus on performance data and instrumentation for machine learning and systems software. The engineer will define and maintain performance data requirements across sources such as profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry. They will identify and document visibility gaps, work with partner teams to address them, guide the development of low-overhead instrumentation and annotation APIs, and safeguard data quality at ingestion. The role also involves measuring and managing instrumentation overhead on real workloads and coordinating progress and commitments across multiple teams.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
At least 8 years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
Proficiency in CUDA.
Working knowledge of machine learning frameworks such as PyTorch or JAX and annotation or tracing APIs such as NVTX.
Experience maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
Experience measuring instrumentation overhead and balancing overhead, fidelity, sampling, and aggregation.
A track record of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and following through to delivery.
Strong debugging skills across component boundaries and ability to validate changes through code review, tests, and independent measurements, including with automated development tools.
Excellent communication skills, particularly in conveying evidence, uncertainty, and next steps to engineers and data users.
Proficiency in Python and C++.

Nice To Haves

Experience establishing telemetry conventions or schemas adopted by other teams.
Background in storage, checkpointing, or data-loading performance.
Expertise in crafting low-overhead event collection.
Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.

Responsibilities

Define and maintain performance data requirements across profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
Identify visibility gaps and demonstrate with concrete evidence which diagnoses cannot be made using current data.
Collaborate with relevant teams to address visibility gaps or develop solutions.
Inform the development of low-overhead instrumentation and annotation APIs, ensuring they convey necessary information efficiently.
Implement data validation, sanitization, checks, and alerting at ingestion to detect gaps or errors before they affect users or automated analysis.
Measure and manage instrumentation overhead and observer effects on real workloads, keeping them within acceptable limits.
Collaborate with multiple partner teams, manage commitments, and ensure continuous progress.