HPC Infrastructure Engineer - GPU Clusters

ElevenLabs

2d•Remote

About The Position

ElevenLabs is hiring an HPC Infrastructure Engineer to own and improve the NVIDIA GPU clusters that power model training. The role is part of a small research infrastructure team and is focused on making compute fast, reliable, and easy for researchers to use. The engineer’s impact is measured through training throughput and researcher velocity. This is a hands-on builder-operator role with broad scope and autonomy. Work includes automating fleet operations, benchmarking GPU providers and networking, and bringing new hardware online, including occasional on-site datacenter work. The role is remote and can be performed globally, with the option to work from ElevenLabs offices in London, New York, San Francisco, or Warsaw.

Requirements

Have operated large-scale Linux server or GPU environments in production and enjoy both building and operating them.
Know the NVIDIA stack, including drivers, CUDA, NCCL, and DCGM, or have deep systems experience and the ability to learn hardware stacks quickly.
Be comfortable with bare-metal environments, server hardware, and high-speed networking.
Write automation in Python and/or Bash and use infrastructure-as-code tools such as Ansible or Terraform.
Be comfortable analyzing metrics, logs, and PromQL to diagnose problems.
Be willing to own substantial scope end to end and serve as a reliable point of contact.
Be willing to handle all kinds of tasks, including datacenter trips.

Nice To Haves

Experience supporting ML training workloads from the infrastructure side, including distributed training failure modes and checkpointing patterns.
Experience evaluating and working with GPU cloud providers.
Experience with parallel filesystems such as WEKA or VAST, or large-scale object storage.
Experience with BMC/IPMI/Redfish automation or PXE provisioning at scale.
Awareness of power and cooling needs for dense GPU deployments.

Responsibilities

Operate and improve the GPU fleet end to end, including provisioning, scheduling, monitoring, upgrades, and capacity planning.
Build automation for fleet health, including node health checks, automated draining and remediation, and burn-in pipelines for new capacity.
Own the software and systems stack beneath training code, including OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, and InfiniBand or RoCE networking.
Run and tune job scheduling systems such as Slurm so researchers receive compute fairly and quickly.
Build and maintain high-performance storage for datasets and checkpoints.
Investigate and resolve performance issues such as stragglers, degraded links, thermal issues, and flaky GPUs, addressing underlying classes of problems.
Evaluate rented GPU capacity by benchmarking and validating it, and hold providers to their SLAs.
Perform hands-on hardware work when needed, including racking, cabling, and diagnostics, and coordinate with datacenter staff and vendors.
Keep clusters secure by default through access control, network isolation, and secrets management.