Senior Systems Software Engineer, GPU Fleet Telemetry

NVIDIA•Santa Clara, CA

1d•$152,000 - $287,500

About The Position

NVIDIA is seeking a Senior Software Engineer to join its DGX Cloud / Fleet Intelligence team. The role develops agent-side systems that collect GPU health, host telemetry, inventory, attestation evidence, and BMC data across NVIDIA’s GPU fleet. It focuses on in-band agent and out-of-band collector systems supporting GPU health monitoring, host and BMC telemetry, diagnostics, and cloud operations automation. The engineer will also contribute to open source software used by developers, operators, and customers working with NVIDIA GPU infrastructure.

Requirements

At least 5 years of industry software engineering experience and a Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience.
Strong Go development experience for Linux services, CLIs, agents, or daemons.
Rust experience, or strong willingness to work in Rust, for out-of-band collector development.
Strong Linux systems knowledge, including processes, filesystems, networking, service lifecycle, logs, permissions, and host diagnostics.
Experience with telemetry, observability, health monitoring, or fleet-management systems, and with Docker, Kubernetes, Helm, and production deployment workflows.
Experience contributing to open source projects or working in public repositories with code review, issue tracking, documentation, release notes, and signed commits.
Familiarity with secure credential handling, enrollment flows, tokens/JWTs, and service-to-service authentication.
Strong debugging skills across hardware-adjacent software, operating systems, containers, and distributed backend integrations.

Nice To Haves

Background with NVIDIA datacenter GPUs, DGX systems, DCGM, NVML, CUDA, GPU drivers, XID/SXID events, or GPU diagnostics.
Experience building host agents, node agents, collectors, monitoring daemons, or Kubernetes DaemonSets.
Background with Redfish, BMCs, firmware inventory, secure boot state, PCIe device inventory, or hardware attestation.
Experience with OpenTelemetry, Prometheus, OTLP gateways, or metric/log export pipelines, and operating software in AI, HPC, cloud, or large-scale datacenter environments.
Meaningful open source contributions in systems software, observability, Kubernetes, Linux, hardware telemetry, Rust, or Go ecosystems.

Responsibilities

Design and develop Fleet Intelligence agent software for Linux hosts, bare-metal systems, and Kubernetes GPU nodes.
Build telemetry and health collection covering NVIDIA GPUs, DCGM/NVML, drivers, CUDA runtime, InfiniBand, containers, kernel and OS state, CPU, memory, disk, and networking.
Develop inventory, enrollment, node identity, local state, attestation, and backend export workflows.
Maintain local API, Prometheus metrics, file export, and OTLP/HTTP export paths.
Build and improve Kubernetes DaemonSet deployment, systemd service packaging, .deb/.rpm packaging, and container image workflows.
Contribute code, tests, documentation, release artifacts, and community-facing engineering practices to open source Fleet Intelligence agent and collector software.
Contribute to out-of-band collection using Redfish/BMC interfaces for inventory, GPU attestation, BMC metrics, and logs.
Improve collector concurrency, rate limiting, retry behavior, credential handling, partial-failure handling, and backend submission semantics.
Collaborate with backend, infrastructure, SRE, security, and datacenter operations teams to deliver reliable GPU fleet observability.