Systems Administrator Lead (Night Shift)

5C Data Centers USA, Inc.•Springfield, OH

75d•$115,000 - $145,000•Remote

About The Position

As a Systems Administrator Lead covering the night shift, you will support, maintain, troubleshoot, and optimize large-scale HPC and AI infrastructure across data center and cloud environments. The role primarily focuses on infrastructure and hardware, including Linux operating systems, GPU servers, high-speed networking, storage connectivity, firmware, drivers, and out-of-band management. It provides overnight coverage to resolve infrastructure issues outside standard business hours. You will serve as a senior technical escalation point for complex incidents involving GPU clusters, compute nodes, networking, storage, and supporting management services. You will collaborate with Network Engineering, Data Center Operations, Deployment Engineering, vendors, and customer technical teams to restore service and improve production HPC reliability. The position is remote in Ohio and works a fixed night shift from 1:00 a.m. to 10:00 a.m. EST, with scheduled on-call escalation.

Requirements

Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
At least 7 years of experience in Linux systems administration, HPC/AI infrastructure, or data center operations, including hands-on production troubleshooting.
Hands-on experience supporting enterprise server hardware, bare-metal compute systems, and large-scale NVIDIA GPU infrastructure or accelerated computing platforms.
Strong understanding of server hardware, firmware, BIOS, BMC, PCIe, NUMA, memory, storage, and network interfaces; experience diagnosing failures using OS logs, BMC logs, and vendor diagnostics.
Experience with NVIDIA drivers, CUDA compatibility, and GPU management tools such as DCGM, Fabric Manager, or nvidia-smi.
Working knowledge of high-performance networking, including InfiniBand, RDMA, RoCE, and NVIDIA/Mellanox adapters.
Experience with out-of-band management tools such as Redfish, IPMI, iDRAC, or iLO.
Proficiency in one or more automation or scripting languages: Python, Bash, or Ansible.
Familiarity with observability platforms such as Prometheus, Grafana, or Datadog.
Ability to troubleshoot across hardware, firmware, operating system, network, storage, and cluster-management layers, supported by strong documentation and communication skills.
Willingness to work a fixed, recurring shift and participate in scheduled on-call escalation.

Responsibilities

Administer, maintain, and troubleshoot Linux-based HPC and AI compute environments, including large-scale GPU clusters using NVIDIA HGX, DGX, or equivalent platforms.
Diagnose hardware, operating system, driver, firmware, networking, and storage failures; perform root-cause analysis and develop corrective actions for recurring issues.
Act as a senior escalation point for complex production infrastructure incidents and restore service to degraded or unstable compute and GPU nodes.
Support bare-metal, virtualized, containerized, and cloud-hosted infrastructure; maintain operational documentation, runbooks, and shift-handoff notes.
Install, configure, validate, and troubleshoot NVIDIA GPU drivers, CUDA components, firmware, and supporting software using tools such as nvidia-smi, DCGM, and NVIDIA Fabric Manager.
Investigate GPU Xid errors, NVLink/NVSwitch faults, PCIe errors, ECC events, GPU resets, and thermal or power-related issues.
Diagnose GPU communication and performance issues involving NCCL, GPUDirect RDMA, PCIe topology, NUMA placement, and multi-GPU systems; validate GPU-to-GPU, GPU-to-network, and GPU-to-storage communication after repairs.
Coordinate replacement and RMA activities for GPUs, GPU trays, baseboards, NVSwitch components, system boards, and NICs.
Administer enterprise Linux distributions and troubleshoot boot failures, kernel issues, systemd services, filesystems, and operating system performance.
Analyze system and kernel logs; manage kernel modules, device drivers, DKMS packages, and operating system updates.
Support Linux networking, storage mounts, authentication, and security controls; investigate OS crashes, kernel panics, and out-of-memory conditions.
Support enterprise server platforms and troubleshoot processors, memory, PCIe devices, NICs, storage controllers, and other hardware components.
Manage BIOS, BMC, CPLD, NIC, GPU, switch, and drive firmware; perform remote console troubleshooting, power cycling, and log bundle generation.
Partner with on-site Data Center technicians on component replacement, cabling validation, break-fix work, and post-repair testing.
Troubleshoot Ethernet, InfiniBand, and RoCE connectivity, including link state, VLAN/bonding issues, MTU mismatches, packet loss, and RDMA/GPUDirect RDMA connectivity.
Collaborate with Deployment Engineering to troubleshoot fabric instability, link degradation, congestion, and node-level connectivity issues.
Troubleshoot server connectivity to parallel, distributed, and enterprise storage platforms; diagnose mount failures, throughput degradation, and capacity issues.
Validate storage network paths and coordinate with storage and networking teams during service-impacting events.
Lead or support incident response for critical infrastructure events, participate in technical bridges, and provide clear outage status updates.
Create incident timelines, root-cause analyses, and corrective-action plans, and maintain accurate records in Jira Service Management or similar ITSM platforms.
Mentor junior and mid-level Systems Administrators and HPC Engineers, providing technical guidance during complex troubleshooting and infrastructure changes.
Promote consistent troubleshooting methods, documentation standards, and operational discipline across shift handoffs.
Participate in scheduled on-call escalation.