Senior Site Reliability Engineer, Observability

NVIDIA•Austin, CA

11d

About The Position

NVIDIA sits at the center of the AI revolution, and the teams behind our data and observability platforms keep the whole engine running! We’re hiring Site Reliability Engineers who want to work on the systems that power everything from large-scale data pipelines to model training clusters to real-time decision making. This isn’t a typical SRE role, you’ll help design and run NVIDIA’s global telemetry backbone, the platform that carries metrics, logs, traces, and profiling data for some of the most demanding workloads in the world. You’ll shape how our AI and data systems are built, set reliability standards, and solve scaling challenges that come with operating at NVIDIA’s pace and scale. If you enjoy diving into distributed systems, building automation that eliminates toil, and partnering with infra and application teams to raise the reliability bar, this is a place where your work will have real impact. And you’ll be joining a group that values curiosity, learning, and blameless engineering, giving you room to grow while working on problems that matter.

Requirements

Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience.
10+ years operating large-scale production systems in roles such as SRE, Production Engineer, or Platform Engineer and 5+ years designing, building, and running observability platforms at scale.
Deep hands-on experience with open-source observability stacks, including Prometheus/Thanos/Mimir for metrics, Loki or Elasticsearch/OpenSearch for logs, and Tempo/Jaeger/OpenTelemetry for tracing and profiling.
Strong programming ability in Python and Go, with Java experience considered a plus.
Solid grounding in Linux internals, networking, storage systems, distributed systems, concurrency, and performance engineering.
Experience architecting multi-region, multi-tenant telemetry pipelines with high availability and strong durability guarantees.
Proven skill in optimizing PromQL, LogQL, trace queries, ingestion paths, indexing strategies, and retention policies.
Strong understanding of SLOs, SLIs, error budgets, incident response, and the operational processes that support reliable systems.
Ability to analyze complex distributed systems, pinpoint failure modes, and drive data-informed debugging and root cause analysis.
Clear communicator who can collaborate effectively across product, platform, infrastructure, and application engineering teams.

Nice To Haves

Designed or led the architecture of a global observability platform supporting thousands of services with strict reliability and performance requirements.
Contributed meaningfully to OpenTelemetry, Prometheus, Grafana, or other major observability open-source projects.
Built high-throughput ingestion pipelines and long-term storage systems, with a strong focus on cost efficiency, retention strategy, and query performance.
Specialized in high-cardinality telemetry, multi-tenant isolation, and advanced retention or tiered storage models.
Worked hands-on with Kafka, Spark, Flink, or large-scale collectors in ultra-high-scale production environments where observability is mission critical.

Responsibilities

Architecting and operating large-scale observability systems that span global regions and support AI, data, and platform services.
Designing resilient pipelines for metrics, logs, traces, profiling, and events that keep critical systems visible and debuggable.
Working closely with platform, infrastructure, and application teams to establish telemetry standards, instrumentation patterns, and integration workflows.
Automating deployments, scaling workflows, and maintenance tasks to cut down toil and level up operational maturity across the stack.
Defining and maintaining SLOs, SLIs, error budgets, dashboards, and alerting models that guide reliability decisions company-wide.
Building self-service tooling and frameworks that make observability easy to adopt for engineers across NVIDIA.
Studying real system behavior to uncover bottlenecks, scaling limits, failure modes, and long-term architecture risks.
Running day-to-day operations including upgrades, performance tuning, break/fix, and rotations that keep the platform healthy.
Leading incident response and root-cause investigations, then driving the follow-through to eliminate repeat failures.
Guiding engineers through design reviews, operational best practices, and reliability-focused decision making.