ML Cluster Operations Engineer

TensorWave•Las Vegas, NV

39d

About The Position

At TensorWave, we’re leading the charge in AI compute, building a versatile cloud platform that’s driving the next generation of AI innovation. We’re focused on creating a foundation that empowers cutting-edge advancements in intelligent computing, pushing the boundaries of what’s possible in the AI landscape. About the Role: We are seeking an exceptional Machine Learning Engineer who has made training and AI workload scheduling a specialty. This is a senior-level role for someone who has significant experience managing distributed machine learning workloads at scale using Slurm and/or Kubernetes. As a technical visionary and hands-on expert, you will lead the evolution of our managed Slurm and Kubernetes offerings, as well as internal health checking and cluster automation.

Requirements

5+ years of experience in cloud infrastructure, HPC, or machine learning roles.
Significant hands-on experience with Slurm in production HPC/ML environments, including understanding of setup/configuration, enroot (pyxis), modules, and MPI.
Strong knowledge of distributed ML languages and frameworks, such as Python, PyTorch, Megatron, c10d, MPI, etc.
Understanding of node lifecycle, including health checks, prolog / epilog scripts, and draining.
Deep understanding of security, compliance, and resilience in containerized workloads.

Nice To Haves

3+ years of hands-on Kubernetes experience, including deep knowledge of the Kubernetes API, internals, networking, and storage.
Proficiency in writing Kubernetes manifests, Helm charts, and managing releases.
Experience with DAGs using K8s native tools such as Argo Workflows.
Foundation in networking, especially as it pertains to RDMA, RoCE, and Infiniband.
Experience with low level kernel libraries, such as CUDA and Composable Kernel.
Contributions to open-source projects or ML/AI tooling.

Responsibilities

Manage and iterate our containerized Slurm (Slurm-in-Kubernetes) solution, including customer configuration and deployment.
Work closely with our engineering team to develop and maintain CI and automation for managed offerings.
Ensure healthy cluster operations and uptime by implementing active and passive health checks, including automated node draining and triage.
Help profile and debug distributed workloads, from small inference jobs to cluster-wide training.
Establish best practices for running jobs at scale, including monitoring, checkpointing, etc.
Mentor and upskill ML engineers in best practices.