AI Infrastructure Engineer

Mercor•San Francisco, CA

2d•$200,000 - $400,000•Onsite

About The Position

Mercor is hiring an AI Infrastructure Engineer to build and operate the GPU and accelerator platform that supports its AI workloads. The position covers clusters and schedulers, training and inference serving, and observability and cost data for measuring fleet utilization. The engineer writes production code daily and owns systems end to end, from design through on-call. Mercor is a profitable Series C company valued at $10 billion. The role is based in person five days a week in Mercor’s San Francisco, New York City, or London offices. The company describes its mission as organizing human intelligence to power the AI economy and building infrastructure that connects human expertise with frontier AI models.

Requirements

At least 5 years building and operating production distributed systems or ML infrastructure; at least 7 years for Staff-level candidates.
Production-quality Python and proficiency in at least one of Go or C++.
Shell scripting for system administration.
Deep hands-on experience with Kubernetes and containers, and working knowledge of Slurm or another HPC resource manager.
Experience running GPU workloads at scale, including scheduling, orchestration, and troubleshooting.
Hands-on experience with at least one distributed training or serving stack: PyTorch Distributed, Ray, vLLM, or Megatron-LM.
Deep experience with at least one major cloud provider.
Experience with observability tools such as DataDog, Prometheus, PromQL, and Grafana.
Comfort with on-call responsibility for critical systems.
Ability to own projects end to end and gather requirements with little direction.

Nice To Haves

Familiarity with or willingness to develop deep expertise in LLM inference optimization, including KV cache systems, disaggregated serving, quantization, and speculative decoding.
Familiarity with or willingness to develop deep expertise in high-performance networking, including RDMA, InfiniBand, NCCL, and NVLink.
Familiarity with or willingness to develop deep expertise in GPU profiling with Nsight or similar tools, and CUDA or Triton kernel work.
Familiarity with or willingness to develop deep expertise in accelerators beyond NVIDIA, including AMD MI-series, TPU, and Trainium.
Familiarity with or willingness to develop deep expertise in capacity planning, billing normalization, or cost attribution across providers.
Familiarity with or willingness to develop deep expertise in automated fault diagnosis, log mining, or fleet-scale AIOps.
Familiarity with or willingness to develop deep expertise in configuration management such as Ansible or Puppet, and secure or air-gapped deployments.

Responsibilities

Design, deploy, and operate GPU clusters across AWS, GCP, other hyperscalers and neoclouds, and on-premise data centers, including custom Kubernetes and Slurm operators and supporting metadata services.
Build accelerator scheduling capabilities, including gang scheduling for distributed jobs, bin-packing, quotas, and fair share for multi-tenant teams.
Run distributed training infrastructure using PyTorch Distributed, Ray, or Megatron, with fast checkpointing and model loading.
Deploy and tune high-throughput LLM serving using vLLM or similar, including tensor and model/expert parallelism, prefill/decode disaggregation, KV cache offload, and FP8/FP4 quantization.
Build benchmarking and traffic-replay tools to measure TTFT, TPOT, and latency percentiles, and set performance baselines for each configuration.
Build health checks across hardware, network, and software layers, and automated remediation that handles fault storms without human intervention.
Instrument fleet utilization with DCGM, Prometheus, and Grafana; write recording rules and alerts used by teams.
Ingest utilization, occupancy, and billing data into a warehouse, attribute spend to teams and workloads, and support capacity planning.
Drive autoscaling, rightsizing, and idle-capacity recovery programs with measurable cost results.
Write runbooks and documentation, and explain infrastructure tradeoffs to researchers, finance, and leadership.
Build internal tools or dashboards used daily by researchers and non-engineers.
Own systems and projects end to end, gather requirements with little direction, and participate in on-call for critical systems.