Network Automation & Reliability Engineer

Ontrac Solutions•Chicago, IL

1d•Remote

About The Position

Ontrac Solutions is seeking a Network Automation & Reliability Engineer to operate and automate hyperscale data center, backbone, and out-of-band (OOB) networks. This is a Python-first role: you will spend more of your week writing code that operates the network than typing on a CLI. You will build the tooling, telemetry pipelines, and self-healing automation that keep a global fleet of data centers and POP sites running, and you will carry the on-call pager for the systems you build. We are looking for an engineer who is genuinely production-grade on both sides of the job — deep routing and switching fundamentals and real, sustained Python software work. Candidates who are strong on one side and thin on the other will not clear screening.

Requirements

Demonstrated, sustained Python development in a production network or infrastructure environment. You have written and maintained tooling that other engineers depended on: config generation and validation, API integrations, telemetry collectors, automated remediation, or test harnesses. You are comfortable with modules, packaging, testing, code review, and version control — not just single-file scripts.
Production experience with BGP (policy, path selection, multihoming), plus IS-IS or OSPF, ECMP, and VXLAN/EVPN or MPLS overlays.
Hands-on operations across at least two of Arista EOS, Juniper Junos (QFX/SRX/PTX/MX), or Cisco IOS-XR / NX-OS.
Ansible and Jinja2, plus NETCONF/YANG, RESTCONF, or vendor REST APIs for model-driven configuration management.
gNMI/gRPC streaming telemetry, OpenConfig models, SNMP, flow telemetry, and dashboarding/alerting on top of them.
Comfortable operating on Linux hosts — networking stack, packet capture, systemd services, and shell scripting.
Git-based workflows with peer review; experience shipping network changes through a pipeline (Jenkins, GitLab CI, or GitHub Actions).
Experience holding a 24x7 rotation for a live network, including incident command and root cause analysis.
Roughly 2–5 years in a network production, network reliability, or network automation role.
Must be located in the United States and authorized to work in the US.

Nice To Haves

Out-of-band network experience — console server fleets (ZPE Nodegrid, OpenGear), RS-232 configuration management, or OOB build-out for new data center capacity.
Data center or POP build-out and turn-up: new product introduction (NPI) for switching platforms, port channelization and optics selection, cabling and rack density planning, or hardware qualification and stress testing.
MACsec, IPsec at scale, or zero-trust segmentation with 802.1X and NAC.
Optical or transport exposure — DWDM, Ciena, PON/OLT/ONU, or IXIA/Spirent test automation.
Building or contributing to a network source of truth (NetBox or in-house) aggregating BGP, link-state, and drain-state data.
Applying LLM or agentic tooling to on-call workflows — automated triage, runbook execution, or incident summarization.
A master's degree in Network Engineering, Telecommunications, or Computer Science.

Responsibilities

Design, write, and maintain Python tooling that provisions, validates, and audits network devices across the global fleet.
Build model-driven provisioning libraries using Jinja2 templating with NETCONF/YANG or REST APIs to codify golden configurations and eliminate configuration drift.
Develop automated remediation for recurring failure classes — route flaps, hardware faults, optical degradation — triggered by syslog and telemetry.
Reduce operational toil: identify manual runbook steps that recur, and replace them with tested, reviewed code.
Operate and scale data center fabrics, backbone links, and out-of-band management networks across a global footprint of data centers and POP sites.
Design and tune BGP policy — local preference, MED, communities, import/export policy, default-route propagation — to control path selection and eliminate single-carrier points of failure.
Support capacity expansion and site turn-ups: high-level design, port channelization, optics and cabling standards, and hardware qualification.
Execute zero-downtime change management in production, including hitless migrations and staged rollbacks.
Operationalize multi-vendor streaming telemetry (gNMI/gRPC, OpenConfig) and tune subscription jobs for signal quality and management-plane efficiency.
Build and maintain observability that supports hop-by-hop path tracing, multi-layer fault isolation, and fast root cause analysis.
Contribute to a real-time network source of truth aggregating BGP, link-state, and drain-state data.
Participate in a 24x7 on-call rotation; lead incident response, write RCAs, and drive the follow-up automation that prevents recurrence.
Maintain and optimize firewall and ACL policy across multi-vendor platforms, keeping rule sets scoped and performant.
Support SIRT/PSIRT CVE remediation through automated regression testing and config-as-code pipelines.
Author and maintain technical documentation — designs, runbooks, and API contracts — for the tooling and networks you own.