Principal Site Reliability Engineer (Vehicle Engineering)

SpaceX • Bastrop, TX

4h

About The Position

SpaceX HPC is a shared compute platform supporting work across the company, including vehicle and structures simulation, machine learning, AI inference, and other workloads. The Principal Site Reliability Engineer will establish and operate an SRE model for these capabilities, emphasizing toil reduction, automation, observability, and sustainable incident response.

The role owns the broader HPC ecosystem as a product, including Linux machines, infrastructure as code, storage, and user-facing applications. The engineer will work alongside HPC systems engineers who design and commission clusters, improving reliability and delivering services that help SpaceX engineers get their work done. A prior HPC job title or aerospace experience is not required; production infrastructure experience and ownership are important.

Requirements

• Bachelor's degree in computer science, engineering, math, or a scientific discipline, or at least 2 years of professional experience operating production infrastructure in lieu of a degree.
• At least 8 years of experience with Linux operating systems in production.
• At least 8 years of experience operating production infrastructure, including servers, services, or networks, and monitoring, debugging, and repairing what you own.
• Must be able to participate in an on-call rotation.
• Must be willing to work extended hours and weekends as needed for incidents, cluster bring-up, and time-critical failures.
• Must be eligible for access to classified material up to TS/SCI with polygraph.
• Must meet U.S. export-control eligibility requirements: be a U.S. citizen or national, lawful permanent resident, refugee, or asylee, or be eligible to obtain required U.S. Department of State authorizations.

Nice To Haves

• At least 8 years of professional experience in SRE, DevOps, or production infrastructure engineering.
• Experience with monitoring and alerting systems such as Prometheus, Grafana, or Nagios.
• Experience deploying and maintaining configuration management or infrastructure-as-code tools such as Ansible, Puppet, or Terraform.
• Experience writing scripts or code, such as Python, to automate common tasks.
• Experience with containers such as Docker, Podman, or Singularity/Apptainer.
• Experience administering Kubernetes for on-premise deployment.
• Experience with distributed or high-performance storage, such as VAST, including capacity, performance, and lifecycle management.
• Familiarity with HPC clusters, schedulers such as Slurm, PBS, or LSF, or GPU compute; this is not required and can be learned.
• Familiarity with scientific computing such as CFD or FEA, ML training workloads such as PyTorch, TensorFlow, or CUDA, or AI inference workloads.
• Good understanding of version control, testing, continuous integration, build, deployment, and monitoring.
• Ability to communicate clearly with users, peers, and vendors in incident and design settings.
• Comfort working with mission-critical and sensitive systems, with an appropriate sense of urgency.

Responsibilities

• Participate in the team's on-call rotation and practice sustainable incident response and blameless postmortems.
• Manage node lifecycle with infrastructure as code, including OS images, firmware, configuration management, and the kernel and driver stack.
• Build observability for HPC administrators and end users, covering cluster, node, and storage health as well as job- and workflow-level signals.
• Reduce toil through automation, balancing production operations with software development that reduces operational work.
• Sustainably manage compute and storage resources.
• Lead capacity planning with users across the company and translate anticipated needs into concrete compute and storage plans.
• Collaborate with HPC systems engineers and engineers across company disciplines on operable, maintainable infrastructure.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal