AI Site Reliability Engineer-GPU infrastructure

Mega Cloud Lab•San Francisco, CA

1d•Onsite

About The Position

The AI Site Reliability Engineer will improve the availability, performance, and operability of AI platform services and their supporting infrastructure. This senior role combines incident response with reliability engineering across Linux, Kubernetes, GPU systems, storage, and networks. The engineer will turn production evidence into automation, safer changes, and sustained reductions in operational toil. Expected deliverables include service dashboards, SLO definitions, actionable alerting, incident runbooks, and tested operational automation. Success is measured through service objective attainment, fewer recurring incidents and less manual toil, completed corrective actions, and validated recovery capability.

Requirements

7+ years in SRE, production operations, or reliability-focused infrastructure engineering, with hands-on responsibility for distributed services.
Strong troubleshooting across Linux, Kubernetes, networking, storage dependencies, and distributed systems under production pressure.
Experience defining SLIs and SLOs, building alerting and response workflows, and using service evidence to prioritize engineering work.
Strong scripting and automation skills, including tested remediation tooling, configuration management, and integration with delivery pipelines.
Experience with incident management, root cause analysis, post-incident follow-through, and clear stakeholder communication.
Ability to balance urgent service restoration with longer-term reliability improvements and document actionable operational procedures.

Nice To Haves

Experience operating AI platforms, ML infrastructure, GPU-backed inference, distributed training, or large HPC-like service environments.
Familiarity with Prometheus, Grafana, ELK/OpenSearch, Loki, PagerDuty, DCGM, and GPU failure diagnostics.
Experience with error budgets, safe deployment strategies, disaster recovery, and capacity planning across mixed batch and online services.
Preferred certifications: NVIDIA-Certified Professional AI Operations (NCP-AIO), Certified Kubernetes Administrator (CKA), Prometheus Certified Associate (PCA), or NVIDIA-Certified Associate AI Infrastructure and Operations (NCA-AIIO). Practical delivery and troubleshooting experience demonstrating corresponding skills is also expected.

Responsibilities

Define service-level indicators and objectives for inference, platform APIs, orchestration, and relevant batch services; establish actionable alerts and use error budgets to guide reliability work.
Build observability across service, workload, GPU, host, and dependency layers using metrics, logs, traces, events, and health signals, and correlate symptoms across systems.
Participate in the agreed-on-call rotation and lead or support incident triage; restore service, maintain incident timelines, coordinate escalations, and communicate impact clearly.
Diagnose service degradation involving Kubernetes, Linux, scheduling, storage, networking, model endpoints, or job orchestration, distinguishing application faults from infrastructure failures.
Automate routine operations and remediation with Python, Bash, or Go; introduce preventive checks for recurring failures and validate automation before production use.
Conduct capacity and performance analysis for GPU memory, utilization, queues, throughput, and service latency, and work with inference and platform engineers on scaling and bottlenecks.
Review production readiness and change plans, including observability coverage, rollback, recovery, and dependency health.
Execute resilience and recovery tests appropriate to the service.
Lead post-incident reviews, track corrective actions, and maintain runbooks and escalation paths.
Use operational reviews to prioritize recurring risks and reliability improvements.