Site Reliability Engineer

xAI•Memphis, MS

23d

About The Position

SpaceXSI is building Super Intelligence and the full Super Intelligence stack, including Colossus supercomputers, Grok models, and applications such as Grok Bot, Grok Build, Grok Imagine, Grok Voice, and X. The Site Reliability Engineer will focus on campus reliability at the Memphis / Southaven data center campus, connecting compute, network, storage, power, and cooling systems. The role designs campus monitoring and observability, leads technical incident response across disciplines, and develops guardrails to reduce the impact and recurrence of incidents. It calls for calm incident leadership, fleet-scale observability judgment, and the ability to advance reliability across software and facility boundaries. The engineer owns monitoring architecture and signal quality across GPU, host, and facility telemetry; provides SEV command support; leads blameless postmortems and corrective actions; and gates high-risk firmware and power or cooling changes. The role also drives cross-functional reliability projects, works with Hardware Failure Analysis on systemic issues, maintains playbooks and dependency maps, partners with the NOC on runbook quality, and defines error budgets and availability objectives as adopted by the business. Participation in on-call rotations and response to SEV-class events is required.

Requirements

Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field, or equivalent experience.
At least 5 years of experience in site reliability, systems engineering, plant operations, or large-scale production operations; power plants, nuclear power plants, high-performance computing, or data centers are preferred environments.
Proven large-scale incident command experience and calm technical leadership on an SRE/NOC bridge or in a plant control room.
Demonstrated fleet- or campus-scale monitoring and observability design, including alert hygiene, suppression, and signal quality.
Hands-on experience with at least one of Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls such as SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus, or BACnet.
Experience working across at least two of compute, network, storage, power, and cooling/facilities telemetry.
Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control room, or NOC partner.
Proficiency in Python or Bash scripting for automation and analysis, plus general experience in at least one systems language such as C, C++, Java, Go, or Rust.
Excellent problem-solving skills and a data-driven approach to reliability engineering.
Ability to collaborate with NOC, data center operations, hardware failure analysis, and infrastructure engineering teams.

Nice To Haves

Experience in power plants, nuclear power plants, or other high-consequence industrial control environments (ICS/OT).
Current or prior work in data centers or facilities spanning power, compute, cooling, and related plant systems, including CDU/direct liquid cooling.
Experience in AI/ML infrastructure or supercomputing environments, including GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm, or Kubernetes.
Fleet firmware and BMC management experience, including Redfish or IPMI, firmware release validation, and staged rollouts.
Reliability engineering experience involving MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, and vendor RCA/RMA partnership.
Experience with CMMS such as Maximo, commissioning/integrated systems testing (IST), and operational turnover.
Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.
Experience running game days, dependency mapping, and closed-loop corrective action programs.
Familiarity with data center hardware and plant signals, including servers, GPUs, networking, power, and cooling, in addition to software telemetry.
Prior work in a fast-paced startup or technology company like SpaceXSI.

Responsibilities

Own monitoring architecture and signal quality across GPU and host telemetry and facility telemetry, determining what to alert on, suppress, and trust.
Use NOC noise-disposition feedback to drive alert suppression and redesign; treat alert noise as a design failure rather than an operator failure.
Provide SEV command support, including technical incident leadership, NOC bridge coordination, paging and escalation, and timeline and severity hygiene.
Run blameless postmortems and drive corrective actions through completion.
Gate high-risk fleet-scale firmware, BIOS, and BMC rollouts using validation, canaries, and rollback, and govern power and cooling work through SOP/MOP/EOP and disciplined change management.
Lead cross-functional reliability projects across compute, network, storage, and facility signal boundaries, including investigation of grey failures.
Partner with Hardware Failure Analysis to use fleet data, including MTBF/AFR and repeat-RMA trends, to develop monitoring, stop-rules, and vendor RCA inputs.
Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current.
Own runbook quality jointly with the NOC; SRE designs runbooks and the NOC operates and corrects them.
Define error budgets and availability objectives at campus and service boundaries as adopted by the business.
Participate in on-call rotations and incident response for SEV-class events at the Memphis / Southaven data center campus.