Manager, Supercomputer Site Reliability Engineering - Memphis

xAI•Memphis, MS

About The Position

SpaceXSI operates multi-campus, hyperscale GPU infrastructure where cluster availability affects training throughput, customer handover commitments, and fleet capital efficiency. The Manager of Supercomputer Site Reliability Engineering is responsible for compute and infrastructure performance and for significantly increasing the SRE team size. The role owns the outcome of greater than 99% compute uptime, drives unplanned facilities compute losses to zero, ensures efficient allocation of more than 3 GW of site power, upgrades fault-tolerant Megapack failover systems capable of supporting all halls for more than four hours (160 MWhr per hall), and builds robust hardware inventory and repair-velocity systems to eliminate hardware chokepoints. The manager serves as the senior operational authority for SRE and leads an integrated reliability organization focused on cluster and facilities health. The role establishes teams, operational processes, incident response and communication practices, vendor accountability, and measurable performance standards. It also drives hardware root-cause analysis, remediation, system understanding, automation, and continuous improvement in reliability and operational efficiency.

Requirements

Bachelor’s degree and 3+ years of experience working in large-scale operations, with 2+ years leading people leaders of technical teams; alternatively, 10+ years of experience working in large-scale operations, with 5+ years leading people leaders of technical teams.
Willingness to travel frequently to data center locations to support operations across sites.
Physical capability to handle data center tasks, including lifting up to 50 lbs unassisted, standing for long periods, and occasional ladder use.
Must be willing to work extended hours and/or weekends as needed.
Ability to thrive in a dynamic, mission-focused environment with on-call ownership of cluster-impacting events.

Nice To Haves

Proven ability to lead large, multi-site, 24/7 operations organizations in fast-paced, high-responsibility settings.
Deep expertise in server hardware, cluster reliability, and data center technologies, from deployment through lifecycle management.
Experience supporting compute-heavy environments such as AI, machine learning, or high-performance computing at scale.
A track record of owning uptime, Service Level Agreements, or reliability metrics for large compute clusters.
Experience leading site reliability engineering or equivalent reliability-focused teams, including root-cause analysis and procedure ownership.
Strong analytical skills and ability to explain technical concepts clearly to diverse audiences, from technicians to executive and tenant partners.
History of partnering with vendors at scale, driving mean time to repair down, and scaling operations across multiple sites.
Familiarity with tooling and automation such as Jira, Python, and Bash used to monitor cluster health and improve team efficiency.
Enthusiasm for SpaceXAI's mission to accelerate human discovery and unravel the universe.

Responsibilities

Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions.
Lead an integrated Supercompute systems engineering reliability team responsible for reactive and increasingly proactive cluster and facilities health and reliability initiatives.
Build out site reliability engineering sub-teams covering compute hardware failure analysis, software tools, cluster monitoring, and integrated site reliability engineering.
Drive operational superiority.
Define the SRE organizational structure and develop a culture of 24/7 operational excellence.
Limit anomaly response times to physical limits and establish robust proactive health processes.
Define on-call responsibilities, incident management protocols, leadership structure, and communication discipline.
Develop training schemes for competent and disciplined monitoring and response teams.
Establish clear operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners.
Own vendor oversight and accountability mechanisms.
Drive mean time to repair toward physical limits.
Continuously improve operational procedures and analysis documentation mechanisms, with a strong emphasis on intelligent automation.
Lead improvement of documentation workflows.
Ensure efficient signal-to-noise in execution.
Define operational targets, health indicators, alerting standards, and incident severity definitions that support measurable team performance insights and accountability.
Cultivate engineering excellence.
Set high technical standards for hardware root-cause analysis and remediation teams, developing industry-leading hardware failure diagnostic expertise.
Integrate with platform engineering teams to develop sophisticated hardware and system understanding and drive high operational reliability.
Minimize repeat hardware issues and lead automation efforts to eliminate manual overhead for unavoidable remediation needs.
Own the outcomes of greater than 99% compute uptime and zero unplanned facilities compute losses.
Ensure efficient allocation of more than 3 GW of site power and upgrade robust, fault-free Megapack failover systems capable of supporting all halls for more than four hours (160 MWhr per hall).
Drive robust hardware inventory systems and repair velocity to eliminate hardware chokepoints.