Staff Software Engineer - Reliability

Luma AI, Inc.•Palo Alto, CA

40d

About The Position

Luma AI runs on thousands of H100 GPUs across multiple providers and clusters for Training, Data Processing and Inference. Working with the Infrastructure and Research teams, the Staff Software Engineer - Reliability maintains the health of our GPU clusters, developing the monitoring and management tools necessary to maximize their performance.

Requirements

Requires a Master's degree (or foreign equivalent) in Computer Science, Information Technology, Electronic Engineering, or related field of study, plus 2 years of experience in the job offered, Software Engineer, or a related occupation.
CI/CD pipelines and automation using AWS and Kubernetes.
CI/CD using gitlab, docker, Kubernetes.
AWS infrastructure using Terraform, packer.
Certs LifeCycle Management using vault with PKI.
Python to automate every aspect of the pipeline.
Python and Shell Scripts for build automation.
Jenkins servers for continuous integration.

Responsibilities

Collaborate with researchers and engineers to specify the availability, performance, correctness, and efficiency requirements of the current and future versions of our GPU infrastructure. (15%)
Work with multiple GPU cloud providers to scale up, scale down, maintain and monitor our GPUs in many clusters. (20%)
Design and implement solutions to ensure the scalability of our infrastructure to meet rapidly increasing demands. (15%)
Implement and manage monitoring systems to proactively identify issues and anomalies in our production environment. (10%)
Implement fault-tolerant and resilient design patterns to minimize service disruptions. (10%)
Build and maintain automation tools to streamline repetitive tasks and improve system reliability. (15%)
Participate in an on-call rotation to respond to critical incidents and ensure 24/7 system availability alongside other infrastructure developers. (5%)
Develop and maintain service level objectives (SLOs) and service level indicators (SLIs) to measure and ensure system reliability. (10%)

Stand Out From the Crowd

Upload your resume and get instant feedback on how well it matches this job.

Upload and Match Resume