AI Infrastructure Engineer (Senior/Staff)

Recruiting From Scratch•San Francisco, CA

1d•Onsite

About The Position

The AI Infrastructure Engineer will join a small, senior GPU platform team under the VP of Engineering and own the clusters, schedulers, and serving stack that the company’s model training and inference depend on. The platform is not yet established, so the engineer will design it, write production code daily, and participate in on-call operations. The role covers GPU clusters across AWS, GCP, neoclouds, and on-premises environments; custom Kubernetes and Slurm operators and their supporting metadata services; accelerator scheduling; distributed training infrastructure; high-throughput LLM serving; fleet health and automated remediation; and utilization, billing, and capacity-planning systems. The position is an individual-contributor role for an experienced engineer who remains hands-on in production code.

Requirements

At least 5 years building and operating production distributed systems or ML infrastructure; at least 7 years for Staff, counted from full-time industry roles only.
Prior experience running GPU workloads at real scale in production.
Demonstrated ownership of at least one GPU cluster or scheduler, distributed training infrastructure, or LLM serving stack.
Production Python experience plus Go or C++.
Deep Kubernetes experience.
Working knowledge of Slurm or another HPC resource manager.
In-depth experience with at least one major cloud provider.
Comfort with on-call responsibilities.
Individual-contributor orientation and continued hands-on coding experience.

Responsibilities

Design, build, and operate GPU clusters across AWS, GCP, neoclouds, and on-premises environments.
Develop custom Kubernetes and Slurm operators and the metadata services that support them.
Build scheduling capabilities including gang scheduling, bin-packing, quotas, and fair share for multi-tenant teams.
Develop distributed training infrastructure using technologies such as PyTorch Distributed, Ray, and Megatron, including fast checkpointing.
Build high-throughput LLM serving on vLLM, including tensor and expert parallelism, prefill/decode disaggregation, KV-cache offload, and FP8/FP4.
Monitor fleet hardware, network, and software health and implement automated remediation.
Use DCGM, Prometheus, and Grafana for fleet monitoring.
Ingest utilization and billing data to support cost attribution and capacity planning.
Write production code and participate in on-call operations.