Senior Software Engineer, Observability

NVIDIA•Us, CA

13h

About The Position

NVIDIA's Observability team is seeking a Senior/Staff Engineer to compose and build the next-generation, multi-region observability platform. This platform powers our rapidly expanding AI, Data, and Observability ecosystem, operating at an immense scale: trillions of metrics, hundreds of terabytes of logs, and billions of distributed traces are processed daily across high-performance datacenters and multi-cloud environments. This is a high-impact, architecture-heavy, and code-first role. You will be responsible for architecting, building, and operating NVIDIA’s unified observability stack, which encompasses metrics, logs, traces, profiles, and analytics powered by advanced technology. Your ownership will span the entire telemetry pipeline, including ingestion, storage, query routing, governance, multi-tenant isolation, GPU-accelerated analytics, and real-time insights. You will play a crucial role in crafting NVIDIA's global observability strategy by collaborating with teams with varied strengths including GPU Compute, Distributed Systems, Networking, ML Infra, AI Platform, and Cloud Services to ensure engineers have deep access to system health, performance, and debugging signals.

Requirements

8+ years of experience with distributed systems, with a focus on observability and monitoring systems
BS or MS in EE, ECE, CS, or equivalent experience.
Deep expertise with metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Opensearch), and tracing platforms (Jaeger, Tempo, OpenTelemetry)
Strong programming skills in Go or Python for automation, operators, and custom integrations
Experience running observability platforms on Kubernetes and operating them at scale across multi-datacenter environments
Demonstrated skill in crafting, optimizing, and scaling telemetry pipelines handling high cardinality and high efficiency data
Solid understanding of distributed systems, performance engineering, and debugging complex workloads
Familiarity with service meshes, networking, and workload instrumentation (Envoy, Istio, OpenTelemetry SDKs)
Collaboration skills and the ability to influence engineering teams to adopt observability guidelines

Nice To Haves

Strong background in software design and system architecture with excellent coding skills in Go and Python for building telemetry pipelines and microservices.
Experience operating large-scale, multi-region observability platforms (metrics, logs, traces).
Proven track record to meet strict SLAs for high-throughput, high-cardinality distributed systems.
Deep expertise in data platforms and telemetry ingestion, storage, and querying layers.
Demonstrated success collaborating across teams to drive platform improvements and standardization.

Responsibilities

Designing and operating scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments
Architecting end-to-end observability pipelines, including ingestion, storage, querying, and visualization
Extending monitoring and alerting with Prometheus, Alertmanager, Thanos/Mimir, Grafana, and OpenTelemetry
Building scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Opensearch stacks
Implementing distributed tracing platforms (Tempo, Jaeger, OpenTelemetry) and integrating with service meshes, load balancers, and APIs
Defining and driving adoption of SLOs, SLIs, and error budgets across services and teams
Automating provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python)
Ensuring reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure)
Embedding security guidelines into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls
Mentoring engineers and shaping Nvidia observability strategy and technical roadmap