Senior Director, Network Reliability Engineering

Lambda•Bellevue, WA

6d•$399,000 - $531,000•Hybrid

About The Position

Lambda is hiring a Senior Director of Network Reliability Engineering to own the end-to-end health of its AI cloud network, spanning GPU compute fabrics, data center networks, backbone, and edge. The role is accountable for network availability, time to detect failures, and time to restore service. It combines break-fix operations with development of software that detects, isolates, and repairs failures automatically, with the goal of creating a self-healing network that prevents most failures from reaching customers or requiring human intervention. The leader will oversee network SRE and automation engineering, incident management, and break-fix coordination with data center teams. They will set the multi-year reliability and automation strategy, establish operational practices and reliability measures, and build and develop an organization of engineers and leaders. The role partners with network, software, capacity, data center, supply chain, and vendor teams, and represents reliability work and technical direction to senior leaders, enterprise customers, and partners during major incidents and reviews. The position requires working from Lambda’s Bellevue or San Francisco office four days per week. Tuesday is currently the designated work-from-home day. The role reports to the VP of Cloud and AI Networking.

Requirements

At least 8 years of engineering management experience, including managing managers and directly managing senior individual contributors.
Proven record leading SRE, network reliability, or production operations organizations for large-scale cloud or data center networks with 24/7 availability requirements.
Network or software engineering background, with production systems experience and the ability to engage credibly on architecture, technical tradeoffs, and code quality.
Demonstrated reduction of toil and incident impact through automation, supported by measurable results such as lower time to restore or a higher share of incidents remediated automatically.
Familiarity with designing, building, scaling, and operating large-scale networks, including BGP routing, leaf-spine fabrics, optics and hardware failure modes, and network telemetry and monitoring.
Experience building closed-loop or event-driven remediation systems and safety controls for automation.
Experience running incident management at scale, including incident command, blameless postmortems, and turning lessons into shipped engineering work.
Ability to foster a data-driven, automation-first, high-velocity organization.
Track record of building high-performing teams in fast-moving, technically demanding environments.
Ability to translate ambiguous business and product goals into clear team priorities and executable engineering plans.
Strong judgment about when to go deep technically, delegate, or escalate.
Ability to remain calm and decisive during major incidents and communicate clearly with customers and executives while incidents are in progress.
Excellent written and verbal communication, including producing high-quality written artifacts to support decision-making and approval of problems and solutions.

Nice To Haves

Experience in GPU cloud, HPC, or AI/ML infrastructure environments.
Experience operating InfiniBand or RoCE fabrics at scale and understanding how network faults affect distributed training jobs.
Experience applying ML- or LLM-based tools to operations, such as anomaly detection, alert correlation, automated triage, or root-cause analysis.
Experience with streaming network telemetry at scale, such as gNMI, OpenConfig, or similar.

Responsibilities

Own network reliability across GPU compute fabrics, data centers, backbone, and edge; be accountable for availability, time to detect, and time to restore.
Run break-fix operations at scale, including fault detection, triage, hardware replacement and RMA, optics and cabling remediation, and coordination with Data Center Operations and remote-hands partners.
Lead networking operations excellence and incident management, including on-call, incident command, escalation paths, customer communication, and blameless postmortems that lead to engineering fixes.
Set the multi-year strategy for a self-healing network using automated detection, fault isolation, traffic shifting, capacity draining and restoration, and repair-work creation.
Convert manual break-fix work into durable software; measure operational toil, set reduction targets, and track the share of failures resolved automatically.
Define network SLOs and error budgets with product and customer-facing teams, and use them to prioritize reliability work against feature and capacity work.
Build safe, trustworthy automation with blast-radius limits, staged rollouts, audit trails, and rollback for automated actions, including AI-assisted triage and remediation.
Own operational standards for incident detection, handling, review, measurement, and learning.
Build, structure, and lead the organization; hire, develop, and retain engineers and managers as the network and organization grow.
Manage through leads and senior individual contributors, providing regular one-on-ones, performance feedback, growth planning, and sponsorship of meaningful work.
Partner with Network Engineering, Network Capacity Automation, Software, and Principal Engineers to incorporate reliability requirements into designs and use failure data to inform architecture, hardware, and vendor decisions.
Partner with Data Center Operations, Supply Chain, and vendors on spares strategy, RMA turnaround, and hardware failure trends.
Represent the organization’s work, reliability metrics, and technical direction to senior leadership, enterprise customers, and partners during major incidents and reviews.
Contribute to organization-wide engineering process improvements in planning, shipping, and learning from incidents.