Staff Network Production Engineer, Operations

Crusoe•San Francisco, CA

2d•$215,000 - $260,000

About The Position

Crusoe Cloud is seeking a Staff Network Production Operations Engineer to own production reliability across its global network infrastructure, including edge, backbone, data center fabric, and GPU cluster interconnects. This hands-on production ownership role focuses on incident response, root cause analysis, and operational excellence to keep hyperscale AI infrastructure running reliably. The work directly supports AI workloads running across thousands of GPUs worldwide. The role contributes to defining SLIs and SLOs, improving observability tooling, and building automation that reduces operational toil. The engineer serves as a key escalation point during high-severity network events, contributes to incident response and stakeholder communication, and documents postmortems. They also author and maintain operational standards, build reliability dashboards and alerts with Architecture and SRE teams, and provide technical guidance to Senior engineers.

Requirements

At least 8 years of production network engineering experience focused on operations, incident response, and reliability in large-scale or internet-scale environments.
Strong Python and scripting proficiency, including writing diagnostic tooling, auto-remediation scripts, and automation pipelines from scratch.
Hands-on experience with streaming telemetry, SNMP, NetFlow/sFlow, Grafana, Prometheus, and ThousandEyes, including scripting against their APIs.
Experience operating RDMA/RoCE lossless fabrics for GPU or HPC workloads, including familiarity with PFC, ECN, and DCQCN tuning.
Expert hands-on production knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP in data center environments.
Proficiency with Arista EOS and Juniper Junos platforms in multi-vendor leaf-spine CLOS architectures.
Experience operating large device fleets across multi-region environments with on-call responsibility, including serving as an escalation point during critical events.
Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.

Nice To Haves

Experience with NVIDIA/Mellanox networking platforms in GPU cluster environments.
Familiarity with Kentik or Arbor for traffic analysis and DDoS visibility.
Experience defining or contributing to SLIs and SLOs in partnership with SRE or product teams.
Exposure to operating fleets of 10,000 or more devices in hyperscale or cloud environments.
Background contributing to post-incident learning programs or organization-wide operational excellence initiatives.

Responsibilities

Help own uptime across Crusoe's global edge, backbone, data center, and GPU cluster network.
Build and maintain Python-based tooling to reduce toil, automate common remediation workflows, and accelerate mean time to resolution.
Develop scripts and tooling for detection, triage, and mitigation during high-severity network events; contribute to incident response, stakeholder communication, and postmortem documentation.
Build tooling to surface systemic issues faster during root cause analyses and drive remediation plans through to closure.
Automate workflows on the monitoring stack—including streaming telemetry, SNMP, NetFlow, Kentik, Grafana, Prometheus, and ThousandEyes—to reduce manual triage and surface actionable signals.
Author and maintain runbooks, escalation playbooks, and SOPs, prioritizing scripting and automation over manual processes.
Partner with Architecture and SRE teams to define and track network reliability metrics, and build supporting dashboards and alerting.
Provide technical guidance to Senior engineers, particularly on automation practices and reducing operational toil, and contribute to operational excellence.