Senior Software Engineer - Infrastructure Storage

Lambda•San Francisco, CA

1d•$266,000 - $395,000•Hybrid

About The Position

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU. If you'd like to build the world's best AI cloud, join us. Note: This position requires presence in our San Francisco office location 4 days per week; Lambda’s designated work from home day is currently Tuesday. In the world of distributed AI, raw GPU and CPU horsepower is just a part of the story. High-performance networking and storage are the critical components that enable and unite these systems, making groundbreaking AI training and inference possible. The Lambda Infrastructure Engineering organization forges the foundation of high-performance AI clusters by welding together the latest in AI storage, networking, GPU and CPU hardware. Our expertise lies at the intersection of: High-Performance Distributed Storage Solutions and Protocols: We engineer the protocols and systems that serve massive datasets at the speeds demanded by modern clustered GPUs. Dynamic Networking: We design advanced networks that provide multi-tenant security and intelligent routing without compromising performance, using the latest in AI networking hardware. Compute Virtualization: We enable cutting-edge virtualization and clustering that allows AI researchers and engineers to focus on AI workloads, not AI infrastructure, unleashing the full compute bandwidth of clustered GPUs. About the Role: We are seeking a seasoned Storage Software Engineer with experience designing and deploying various storage protocol solutions at scale (object, block, and file). This is a unique opportunity to work at the intersection of large-scale distributed systems and the rapidly evolving field of artificial intelligence infrastructure. This is an opportunity to have a significant impact on the future of AI. You will be building the foundational infrastructure that powers some of the most advanced AI research and products in the world.

Requirements

10+ years of experience in storage engineering with at least 5+ years in a management or lead role.
Systems-Level Programming and Architecture
Storage Protocol and API Mastery
Storage Performance Optimization
DPKD SPKD
Physical Infrastructure Knowledge
Operational Acumen
Experience in serving one or more of the following storage protocols: object storage (e.g., S3), block storage (e.g., iSCSI), or file storage (e.g., NFS, SMB, Lustre).
Professional individual contributor experience as a storage engineer or storage SRE.
Familiarity with modern storage technologies (e.g., NVMe, RDMA, DPUs) and their role in optimizing performance.

Nice To Haves

Experience driving cross-functional engineering management initiatives (coordinating events, strategic planning, coordinating large projects).
Experience with NVidia SuperNIC DPUs for edge-caching (such as implementing GPUDirect Storage).
Deep experience with Vast, Weka and/or NetApp in an HPC or AI Infrastructure environment.
Deep experience implementing CEPH in an HPC or AI infrastructure environment at a scale greater than 100PB.
Experience driving organizational improvements (processes, systems, etc.)
Experience training, or managing managers.

Responsibilities

Design, develop, and maintain software for storage systems, focusing on performance, scalability, and reliability.
Implement and optimize storage protocol APIs for file (e.g., NFS, SMB), block (e.g., Fibre Channel), and object (e.g., S3) access.
Develop distributed systems for managing and orchestrating storage resources across multiple storage solutions and redundant arrays.
Collaborate with hardware and system architects to integrate software with various storage solutions, including NVMe and GPU-direct storage.
Troubleshoot and debug complex issues in a production data center environment.
Contribute to the full software development lifecycle, from requirements gathering and design to deployment and maintenance.
Work closely with the storage software teams and networking teams to execute on cross-functional infrastructure initiatives and new data-center deployments including integration of storage protocols across a variety of on-prem storage solutions.
Work closely with the control plane and MK8s teams to meet customer/product requirements for usability, reliability, and telemetry.
Work with the observability team to build/track SLOs/SLIs.
Work closely with Networking, Compute, and Storage Software Engineering teams to deploy high-performance distributed storage solutions to serve AI/ML workloads.
Partner with the fleet engineering team to ensure seamless deployment, monitoring, and maintenance of the distributed storage solutions.
Stay current with the latest trends and research into AI and HPC storage technologies.
Work with the Lambda product team to uncover new trends in the AI inference and training product category that will inform emerging storage solutions.
Optimize protocol solutions for the AI product vertical exploring optimizations for AI Inference, training, and scientific computing applications.
Experience building a high-performance team through deliberate hiring, upskilling, planned skills redundancy, performance-management, and expectation setting.