Senior Site Reliability Engineer

Andromeda Cluster • San Francisco, CA

5d • Remote

About The Position

Andromeda is hiring a Senior Site Reliability Engineer to design, operate, and debug large-scale GPU infrastructure for distributed training and inference. The role focuses on production GPU clusters and the failure modes of distributed training, requiring systems-level performance reasoning from network fabric through kernels and frameworks. The engineer will work directly with customers running demanding AI workloads and help build the infrastructure backbone for reliable, scalable AI compute.

The position offers significant ownership and autonomy to shape systems at a foundational level, influence technical direction, and help define world-class AI infrastructure operations. It involves collaboration with customers and compute providers across a platform serving model builders, research labs, and inference providers.

Requirements

• Deep, hands-on experience operating large-scale GPU clusters in production, such as NVIDIA A100, H100, B200, or equivalent.
• Direct understanding of GPU memory hierarchies, ECC behavior, thermal throttling, and hardware failure modes.
• Production experience with InfiniBand, RoCE, or NVLink fabrics for distributed training, including diagnosing collective-operation performance and network congestion.
• Working knowledge of distributed training systems and frameworks such as NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, or FSDP.
• Expert-level Linux knowledge, including kernel tuning, NVIDIA driver and CUDA toolkit management, cgroup and namespace internals, and syscall- and hardware-level performance profiling.
• Strong experience running Kubernetes in production with GPU workloads, including device plugins, topology-aware scheduling, multi-cluster federation, and custom operators; Slurm or other HPC scheduler experience is also valued.
• Strong software engineering skills in Python, Go, or Bash, with experience building production-grade tools and services.
• Infrastructure-as-Code proficiency with Terraform, Helm, Ansible, or an equivalent.
• Hands-on experience building monitoring and alerting for GPU infrastructure, including GPU-specific telemetry such as DCGM, nvidia-smi, and fabric manager metrics integrated into actionable dashboards.
• Proven track record leading incident response for complex distributed systems spanning hardware, firmware, networking, drivers, orchestration, or application code.

Nice To Haves

• Experience with high-performance parallel file systems such as VAST, Weka, Lustre, or GPFS, and with checkpoint I/O and data-loading bottlenecks in large training runs.
• Experience profiling and optimizing distributed training performance, including identifying stragglers, tuning collective communication, improving MFU, and reducing idle GPU time.
• Experience with physical cluster design, including rack layout, power and cooling constraints, network topology, and hardware validation or burn-in at scale.
• Experience leading or mentoring a team of infrastructure engineers.

Responsibilities

• Design and evolve multi-provider, multi-region GPU compute clusters optimized for large-scale training, including topology-aware scheduling, networking, and storage decisions.
• Serve as the primary technical point of contact for customers running large-scale training workloads; onboard, troubleshoot, and optimize their workloads, often in real time.
• Define SLOs and error budgets for GPU infrastructure failure modes and manage capacity planning across heterogeneous GPU fleets.
• Ensure the health and performance of InfiniBand, RoCE, and NVLink interconnects; diagnose and resolve fabric issues affecting distributed training.
• Build visibility into GPU utilization, memory pressure, interconnect throughput, training job performance, and hardware health.
• Develop production-grade automation for cluster provisioning, GPU health checks, job scheduling, self-healing, and firmware and driver lifecycle management.
• Lead incident response for complex failures spanning hardware, networking, orchestration, and ML frameworks; conduct blameless postmortems and drive systemic fixes.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed