Senior Software Enginering - Mk8S/Control Plane
About The Position
Lambda is seeking a Senior Software Engineer to join its Managed Kubernetes (Mk8s) team. This role involves shaping the architecture, reliability, and automation of Kubernetes-based infrastructure for AI workloads on bare metal. The engineer will work on managed orchestration services, including Managed Kubernetes and Managed Slurm on Kubernetes, contributing to systems that are reliable, performant, and simple for customers. The position requires a deep understanding of distributed systems, GPU-accelerated computing, and Cloud Native infrastructure, going beyond just operating Kubernetes to building solutions that consider compute, network, storage, and security interactions. Collaboration with NVIDIA's open-source ecosystem and internal teams is expected to deliver a world-class managed platform.
Requirements
- 6+ years of experience in software engineering, with a track record of owning significant technical scope within a team.
- Deep understanding of Kubernetes internals: controllers, schedulers, operators, CRDs, CSI, CNI, and extension patterns.
- Solid grasp of distributed systems fundamentals: fault tolerance, graceful degradation, and failure handling in large-scale environments.
- Experience operating the control plane and low-level pieces of large-scale Kubernetes clusters.
- Experience with observability at scale: Prometheus, Grafana, distributed tracing, and building actionable alerting systems.
- Strong programming skills in Go and Python; ability to collaborate effectively on shared codebases.
- Solid knowledge of Linux systems, networking, containers, and cloud infrastructure.
- Pride in owning and delivering core components of products and platforms.
Nice To Haves
- Experience building and operating managed Kubernetes services (GKE, EKS, AKS, or similar) or working on Kubernetes control plane components.
- Hands-on experience with NVIDIA's GPU/networking ecosystem: GPU Operator, device plugins, DCGM, MIG, Network Operator, NCCL tuning, or similar.
- Familiarity with HPC and traditional job schedulers (Slurm) and Kubernetes-native batch scheduling (KAI, Volcano, Kueue).
- Familiarity with GPU, InfiniBand, RDMA, or high-performance computing on Kubernetes.
- Exposure to storage architecture for AI/ML workloads.
- Past contributions to CNCF projects or Kubernetes SIGs.
Responsibilities
- Design, build, and maintain scalable control plane services, operators, and custom Kubernetes controllers.
- Develop automation in Go/Python for end-to-end cluster lifecycle management (provisioning, upgrades, patching, deletion).
- Build GPU-aware orchestration systems supporting GPU scheduling and resource allocation.
- Partner with the Network team on networking solutions for AI workloads, including CNI integration, high-performance fabrics, RDMA, and GPUDirect.
- Write resilient systems that handle failure gracefully (timeouts, retries, backoff, degraded-mode operation) in large-scale distributed environments.
- Develop platform services for inference, including model serving infrastructure, autoscaling based on inference load, and multi-model deployment patterns.
- Build internal tools and CLIs for ML/AI teams to deploy and monitor their inference services.
- Support and debug production issues through on-call rotation.
Benefits
- Health, dental, and vision coverage for you and your dependents
- Wellness and commuter stipends for select roles
- 401k Plan with 2% company match (USA employees)
- Flexible paid time off plan
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
