Network Engineer

TekWissen•Huntsville, AL

12d•Onsite

About The Position

Support networks within hardware validation labs around the world, including datacenter fabrics, timing, optical, and provisioning systems that enable accelerator teams to validate new platforms from initial bring-up through production handoff. The role covers L2/L3 lab fabric design, precision timing as a service, intent-based provisioning and automation, optical and test infrastructure, lab operations and reliability, and cost reduction through decommissioning. The team uses internal platforms with direct industry analogues. Transferable experience with Git-backed hierarchical configuration management and Jinja, policy-driven intent automation, graph-modeled network state and diff-before-push workflows, NetBox-style inventory and asset sources of truth, network management and topology visualization, and daily revision-control diff reviews is relevant. Early success includes migrating a rack boundary to routed operation, delivering an intent-driven provisioning flow without manual CLI steps, extending monitored timing service to a new cluster, and tracking and categorizing unplanned support requests.

Requirements

Deep L2/L3 networking knowledge, including VLANs, loop prevention, LAG, IPv4/IPv6 addressing, SLAAC and router advertisements, DHCP relay and helpers, and end-to-end PXE boot flows.
BGP expertise, including eBGP peering design, prefix policy, uplink preference tuning, and per-rack or per-site route origination.
Datacenter fabric experience with leaf/spine design, EVPN/VXLAN overlays, and migration from bridged Layer 2 to routed boundaries.
IPv6 operations experience, including RA Guard, address locking, DHCPv6 relay, MAC-stable allocation behavior, client-identifier pitfalls, and /64-per-rack planning.
PTP (IEEE 1588) design and troubleshooting experience, including grandmaster placement, boundary and transparent clock concepts, and synchronization monitoring against microsecond budgets.
AI/HPC fabric knowledge, including PFC and lossless Ethernet tuning, link training, SerDes and current-generation high-speed optics, and the impact of skew and tail latency on collective communication.
Knowledge of fiber types and connectors, optical circuit switching concepts, commercial traffic generators, cable validation, and fiber characterization using hardware timestamps.
Access-security experience with port ACLs, 802.1X/MAB, RADIUS, quarantine and remediation VLANs, and device onboarding and registration.
Deep experience with at least one datacenter network operating system—Arista, Cisco, Juniper, or SONiC—and ability to learn others, including API-driven operation with rollback.
Packet-level troubleshooting, hardware timestamp analysis, method-of-procedure authorship, maintenance-window execution, and rollback planning.
Python skills for network automation, including device APIs, text templating, input validation, and operator CLI tooling.
Jinja2 template design for reuse across sites and roles, safe defaults, and per-port rendering from inventory.
Distributed revision-control fluency, including branching, review, landing, and reverting changes; configuration review practices; and CI gates for generated configuration.
Declarative source-of-truth and intent-modeling approach in which inventory and intent render device configuration and generated configuration is not hand-edited.
Linux networking experience with interfaces, routing tables, namespaces, capture tooling, service management, and cookbook-style configuration management such as Chef.
Data hygiene practices covering accurate inventory, roles, port maps, serial-to-port bindings, drift detection, and reconciliation loops.
Lab or datacenter operations experience, including on-call rotations, incident response, break/fix prioritization, vendor RMAs, and multi-site coordination.
Change-management discipline, including procedure authorship and peer review, blast-radius control, and pilot-before-scale practices.
Cost-awareness skills involving circuit inventories, carrier coordination for turn-down, recurring-spend tracking, and chargeback confirmation.
Cross-functional delivery experience with platform teams, hardware bring-up teams, and lab customers.

Nice To Haves

AI/HPC cluster networking, including scale-up versus scale-out fabric design, RDMA-capable fabrics, and sensitivity of collective operations to timing.
Experience operating large fiber plants or optical switching.
Experience in IPv6-first or IPv6-only environments at scale.
Observability experience with time-series metrics, topology visualization, and low-noise alert design.
Experience evaluating vendors, switch selection criteria, and optics compatibility testing.

Responsibilities

Design and operate L2/L3 lab fabrics, including VLANs, loop prevention, link aggregation, IP addressing, DHCP relay, and PXE boot flows.
Design BGP peering and prefix policies, tune uplink preference, and originate per-rack or per-site routes toward border and aggregation layers.
Design datacenter leaf/spine fabrics and EVPN/VXLAN overlays, and migrate racks from bridged Layer 2 to routed Layer 3 boundaries.
Operate IPv6 networks, including first-hop security, DHCPv6 relay, address allocation, and per-rack /64 planning.
Design and troubleshoot PTP precision timing, including grandmaster placement, clocking concepts, and synchronization-quality monitoring.
Tune AI/HPC fabrics for lossless Ethernet and assess link training, SerDes, optics, skew, and tail latency effects on collective communication.
Support optical and physical plant infrastructure, including fiber and connectors, optical circuit switching, traffic generators, cable validation, and fiber characterization.
Implement access security using port ACLs, 802.1X/MAB, RADIUS, quarantine and remediation VLANs, and device onboarding and registration.
Operate and automate multi-vendor datacenter network devices, using APIs and rollback procedures.
Troubleshoot network issues at packet level and analyze hardware timestamps; author procedures and plan maintenance windows and rollbacks.
Build Python-based network automation, device API integrations, templates, input validation, and operator CLI tools.
Develop reusable Jinja2 templates with safe defaults and per-port rendering from inventory.
Manage configuration as code through distributed revision control, reviews, reversions, and CI gates for generated configuration.
Use declarative intent and inventory as sources for generated device configuration; do not hand-edit generated configuration.
Work with Linux networking, including interfaces, routing tables, namespaces, capture tools, service management, and Chef-style configuration management.
Maintain inventory accuracy, detect configuration drift, and operate reconciliation loops.
Participate in on-call rotations and incident response; prioritize break/fix work, manage vendor RMAs, and coordinate across sites.
Apply change discipline through procedure authorship and peer review, blast-radius control, pilots, and staged scaling.
Manage circuit inventories and carrier coordination for service turn-down; track recurring spend to zero and confirm chargeback.
Coordinate delivery with platform, provisioning, observability, hardware bring-up, and lab customer teams, including consultative support outside the plan of record.
Track and categorize unplanned support requests.
Migrate a rack boundary to routed operation, deliver an intent-based provisioning flow without manual CLI, and extend monitored timing service to a new cluster.