Manager, Supercomputer Site Reliability Engineering - Memphis

SpaceXSI•Southaven, MS

14h

About The Position

SpaceXSI operates multi-campus, hyperscale GPU infrastructure where cluster availability affects training throughput, customer handover commitments, and fleet capital efficiency. The Manager of Supercomputer Site Reliability Engineering is responsible for compute and infrastructure performance and for significantly expanding the SRE team. The role focuses on continuous site monitoring, cross-discipline incident command, hardware root-cause analysis, software systems that improve repair velocity, and physical infrastructure reliability, with close attention to system margins and resiliency. The manager owns the outcome of greater than 99% compute uptime, drives unplanned facilities-related compute losses to zero, ensures efficient allocation of more than 3 GW of site power, upgrades robust fault-free Megapack failover systems capable of supporting all halls for more than four hours (160 MWh per hall), and builds robust hardware inventory and repair-velocity systems to eliminate hardware bottlenecks. The position leads the SRE organization, establishes operational standards and structures, and partners with engineering, data center operations, vendors, and other service owners.

Requirements

Bachelor’s degree and 3+ years of experience working in large-scale operations, with 2+ years leading people leaders of technical teams; or 10+ years of experience working in large-scale operations, with 5+ years leading people leaders of technical teams.
Willingness to travel frequently to data center locations to support operations across sites.
Physical capability to handle data center tasks, including lifting up to 50 lbs unassisted, standing for long periods, and occasional ladder use.
Willingness to work extended hours and/or weekends as needed.

Nice To Haves

Proven ability to lead large, multi-site, 24/7 operations organizations in fast-paced, high-responsibility settings.
Deep expertise in server hardware, cluster reliability, and data center technologies, from deployment through lifecycle management.
Experience supporting compute-heavy environments such as AI, machine learning, or high-performance computing at scale.
A track record of owning uptime, Service Level Agreements, or reliability metrics for large compute clusters.
Experience leading site reliability engineering or equivalent reliability-focused teams, including root-cause analysis and procedure ownership.
Strong analytical skills and the ability to explain technical concepts clearly to technicians, executives, and tenant partners.
A history of partnering with vendors at scale, reducing mean time to repair, and scaling operations across multiple sites.
Familiarity with monitoring and automation tools such as Jira, Python, and Bash.
Enthusiasm for SpaceXAI's mission to accelerate human discovery and unravel the universe.
Ability to thrive in a dynamic, mission-focused environment with on-call ownership of cluster-impacting events.

Responsibilities

Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions.
Lead an integrated Supercompute systems engineering reliability team responsible for reactive and increasingly proactive cluster and facilities health and reliability initiatives.
Build out SRE sub-teams focused on compute hardware failure analysis, software tools, cluster monitoring, and integrated site reliability engineering.
Drive operational superiority and define the SRE organizational structure and culture of 24/7 operational excellence.
Reduce anomaly response times and establish robust proactive health processes.
Define on-call responsibilities, incident management protocols, leadership structure, and communication discipline.
Develop training programs for monitoring and response teams.
Establish operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners.
Own vendor oversight and accountability mechanisms.
Drive mean time to repair toward physical limits.
Define and continuously improve operational procedures and analysis documentation, emphasizing intelligent automation.
Improve and strengthen documentation workflows.
Ensure efficient signal-to-noise in execution.
Define operational targets, health indicators, alerting standards, and incident severity definitions to support measurable performance insights and accountability.
Drive high technical standards for hardware root-cause analysis and remediation teams.
Integrate with platform engineering teams to develop hardware and system understanding that supports high operational reliability.
Minimize recurring hardware issues and lead automation efforts to eliminate manual work for unavoidable remediation.
Own the outcome of greater than 99% compute uptime.
Drive unplanned facilities compute losses to zero.
Ensure efficient allocation of more than 3 GW of site power.
Upgrade fault-free Megapack failover systems to support all halls for more than four hours (160 MWh per hall).
Drive robust hardware inventory systems and repair velocity to eliminate hardware bottlenecks.
Travel frequently to data center locations to support operations across sites.
Work extended hours and/or weekends as needed.