NOC Manager — AI Infrastructure Operations (24/7)

Inovi•Richmond, VA

10d•$90,000 - $120,000•Onsite

About The Position

We are hiring a hands-on NOC Manager to lead the 24/7 team responsible for monitoring, configuring, and troubleshooting our AI compute infrastructure. This is a working manager role: you will own shift coverage, escalation, and team development, and you will also be at the keyboard yourself — logged into devices, reading interface counters, tracing traffic paths, pushing configuration, and driving major incidents to resolution. This is a remote-hands operating model. The team works from our Richmond office, not the data center floor. All configuration and troubleshooting is done via CLI, out-of-band management, and console access, with physical work executed by on-site smart-hands technicians and colocation staff whom you will direct and hold accountable. The ideal candidate is a network engineer first and a manager second. AI training and inference workloads put unusual pressure on the network — dense east-west traffic, lossless fabric requirements, and jobs that fail expensively when a single link degrades. We need someone who understands what those failures look like at the packet and interface level, and who can build a team that catches them before customers do.

Requirements

7+ years in network engineering or network operations, with 2+ years leading or supervising a technical team (formal management or team lead)
Deep, hands-on expertise with enterprise and data center networking: BGP, OSPF, VLANs, STP, VXLAN/EVPN, QoS, ACLs, LACP, and multicast fundamentals
Strong command of Cisco platforms — Nexus, Catalyst, ASR/ISR, and NX-OS/IOS-XE CLI — in production environments
Demonstrated experience in a 24/7 operational environment with formal incident and change management
Proficiency with monitoring and observability tooling (SolarWinds, Zabbix, LibreNMS, Prometheus/Grafana, Splunk, or equivalent)
Working Linux administration and CLI skills
Experience operating infrastructure remotely: out-of-band management, terminal servers, IPMI/BMC/iDRAC, jump hosts, and console access
Track record of diagnosing hardware and physical-layer failures without physical access, and directing remote-hands technicians to a correct fix
Ability to work on-site at our Richmond, VA office and participate in an on-call rotation
US Citizen or Permanent Resident

Nice To Haves

Cisco certification strongly preferred — CCNP Enterprise or CCNP Data Center; CCIE is a significant differentiator. CCNA plus deep production experience will be considered.
Experience with AI/ML, HPC, or GPU cluster environments
Experience with lossless/low-latency fabrics: InfiniBand, RoCEv2, PFC/ECN tuning, or Cisco ACI
Automation and scripting: Python, Ansible, or equivalent for config management and operational tooling
Familiarity with data center facilities concepts as they surface in remote telemetry — power distribution and redundancy, PDU monitoring, thermal alarming, DCIM tooling
Experience managing infrastructure in colocation or multi-site environments
Exposure to Juniper, Arista, or NVIDIA/Mellanox networking alongside Cisco
ITIL, PMP, or equivalent operational process certification
Experience standing up or scaling a NOC from an early or immature state

Responsibilities

Manage and schedule a 24/7/365 NOC team, including shift rotations, holiday coverage, and on-call escalation paths
Hire, onboard, coach, and develop NOC engineers; conduct performance reviews and build individual growth plans
Establish and enforce shift handoff standards so nothing is dropped between rotations
Serve as the senior escalation point for the NOC, including nights and weekends when severity warrants
Personally configure, troubleshoot, and resolve complex network and infrastructure issues — routing, switching, fabric, transport, and physical-layer symptoms diagnosed remotely
Work directly on production devices via SSH, out-of-band console, and terminal servers; build, validate, and push configuration changes
Diagnose physical-layer faults from remote telemetry — optical power levels, CRC and discard counters, link flaps, transceiver diagnostics — then dispatch and precisely direct remote-hands technicians for cable, optic, or hardware replacement
Act as incident commander for Sev-1/Sev-2 events: coordinate responders, own customer and executive communications, and drive to restoration
Execute and review change management for production network changes, including maintenance windows and rollback planning
Own health monitoring for GPU compute clusters, high-speed interconnect fabric, storage networks, power, and cooling
Tune monitoring, alerting, and observability tooling to reduce noise and catch degradation — flapping links, CRC and discard counters, optical power drift, thermal trends — before workloads are affected
Partner with platform and ML infrastructure teams to correlate job failures with underlying network and hardware events
Build and maintain runbooks, escalation matrices, and standard operating procedures; retire tribal knowledge
Drive post-incident reviews and track corrective actions to completion
Report on MTTA, MTTR, uptime, ticket volume, and recurring problem trends to leadership
Identify repetitive manual work and drive automation of it
Own relationships with colocation providers, carriers, hardware vendors, and smart-hands teams — including dispatch procedures, response-time expectations, RMA workflows, and support-case escalation
Write remote-hands work instructions clear enough for a technician with no context to execute correctly on the first attempt