Senior Staff Network Engineer

Lambda•Bellevue, WA

17d•$324,000 - $480,000•Hybrid

About The Position

Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU. You will be joining a team of software, hardware and network engineers building one of the largest AI training and inference networks in the world. Lambda has 10x'd over the last three years and the network has to keep pace. As a Senior Staff Network Engineer you'll have the opportunity to be across all aspects of the network, including industry leading backend, frontend, backbone and edge networks. You will lead the projects that design, build and scale it. You are the engineer the team routes its hardest problems to, and the one whose designs other engineers follow. Our vision is bold and is not an incremental exercise. We will continually re-evaluate and reinvent our current fabric topology, routing design and the automation underneath all of it, while operating the existing network flawlessly for all customer workloads. This role will report to our Senior Network Engineering leader and will work alongside an existing, highly capable team of network engineers.

Requirements

Have owned the technical direction of a significant area of a production network — you can walk us through a design you made, why, what it cost, and what you'd change
Have led large production-scale networking projects across multiple teams, from design through delivery
Expert in large network designs to achieve maximum availability and performance
Deep expertise in data center, backbone and internet protocols and technology
Experience building and tuning networks for low latency, fast convergence and high availability
Have experience with cloud provider networking (such as AWS, GCP, OCI)
Are comfortable on the Linux command line, and have an understanding of the Linux networking stack and internals
Strong automation skills (Python, Ansible, Jinja), network APIs, and git or similar source control
Production experience with multiple network gear vendors (Arista, Juniper, Cisco, Cumulus/SONiC, Opengear)
Have raised the technical bar of engineers around you through mentorship and design review
Excellent communicator both written and verbal. You leverage high quality written artifacts to inform high quality decision making and get approval on problems and solution opportunities

Nice To Haves

Have knowledge or experience maintaining Software Defined Networks (SDN)
Experience automating network configuration
Hands-on with HPC/AI networking: RoCEv2 and/or InfiniBand (Congestion Control, VLs, partitions), GPUDirect RDMA concepts.
Experience with DWDM technologies and SD-WAN

Responsibilities

Own the architecture of major areas of Lambda's network end to end — you make the design calls, you help build it, and you're connected to how they hold up in production
Lead large networking projects that may span multiple teams, from design through build to turn-up, and drive them to completion
Write the designs, RFCs and reviews other engineers build from, and hold the bar in design review for work across the team
Be the deep technical escalation point for the hardest production problems — the packet path, the routing table, the vendor's firmware, the code
Drive the automation of your domain, replacing manual operational work with software that holds up under scale
Partner with hardware, platform and product teams to resolve technical questions that cross team boundaries
Qualify and integrate new network hardware and software, and hold vendors to account on defects and roadmap
Guide partner teams to automate and scale through robust software systems and services
Guide partner teams build the right telemetry and automation to ensure our network health is observable and self healing
Mentor senior and mid-level engineers, and raise the bar through design review, code review and hiring
Contribute to the multi-year technical strategy for Lambda's network, and own the parts of it that fall in your area
Work with internal and external customers to resolve network related issues
Operate what you build. You'll take part in day 2 operations and the on-call rotation, because the fastest feedback loop between design and reality runs through the pager