Machine Learning Systems Administrator / DevOps Engineer: Training Clusters & Inference Cloud

Zyphra•San Francisco, CA

9d•Onsite

About The Position

As a Machine Learning Systems Administrator / DevOps Engineer, you will maintain and build the core infrastructure behind model training and the inference cloud. You will split your time between large-scale training clusters and production deployments and services running models on heterogeneous compute across multiple accelerator types and vendors. You will work closely with training, inference, and product teams to keep systems reliable, secure, and fast, while laying the groundwork for scalable workflows. You will administer and automate Linux-based training cluster environments, manage user access and security auditing, monitor resources and cluster health, validate new hardware, support developer workflows, and enable large-scale training jobs. For the inference cloud, you will deploy, operate, and scale inference services; manage orchestration and deployment pipelines; build observability; own reliability and incident response; plan capacity and track utilization and costs; and secure production systems. The role owns critical systems end to end and directly affects developer productivity, training throughput, and inference platform performance and reliability.

Requirements

Experience with Linux system administration, user and access management, and automation.
Demonstrated expertise in scripting languages for system tooling and automation, such as Bash or Python.
Experience with containers such as Docker and scheduling or orchestration systems such as Slurm or Kubernetes.
Experience running production services, including monitoring and alerting, deployment pipelines, and incident response.
Experience building tooling for cluster validation and reliability, including accelerator, networking, and storage health checks.
Experience setting up and managing developer tools and third-party services, such as cloud storage providers, Docker Hub, Slack, Gmail, Telegraf, and experiment trackers.
Excellent debugging and troubleshooting skills across compute, storage, and networking.
Strong communication skills and ability to collaborate across technical and non-technical teams.

Nice To Haves

Experience with infrastructure as code, such as Ansible or Terraform.
Prior work supporting ML/AI infrastructure, including GPU management and workload optimization.
Experience deploying and operating LLM serving stacks, such as vLLM, SGLang, or Ray.
Experience with multiple accelerator platforms, such as NVIDIA CUDA, AMD ROCm, or TPUs.
Experience with high-performance networking, such as InfiniBand or RoCE, and distributed storage.
Experience working with cloud platforms such as AWS, Azure, or GCP.
Familiarity with Apptainer and integrating containers into Slurm and Kubernetes.

Responsibilities

Administer and automate Linux-based cluster environments.
Manage user onboarding and offboarding, access control, and security auditing.
Monitor system resources, job scheduling, and cluster health.
Bring up and validate compute, networking, and storage hardware.
Support and improve developer workflows, including VSCode remote development and Docker.
Enable and support large-scale training jobs.
Deploy, operate, and scale inference services across heterogeneous accelerators.
Manage container orchestration, CI/CD, and rollout pipelines for model deployments.
Build production observability, including metrics, logging, alerting, and dashboards.
Own service reliability, uptime, incident response, and postmortems.
Plan capacity, autoscaling, and utilization and cost tracking across hardware types.
Secure production systems through secrets management, network policy, and tenant isolation.
Build tooling for cluster validation and reliability, including accelerator, networking, and storage health checks.
Set up and manage developer tools and third-party services.