Senior Cloud System Administrator

5C Data Centers USA, Inc.•Springfield, OH

3d•$85,000 - $120,000•Remote

About The Position

As a Senior Cloud System Administrator, you will support, maintain, troubleshoot, and optimize large-scale HPC and AI infrastructure across data center and cloud environments. This role is primarily focused on the infrastructure and hardware layer, including Linux operating systems, GPU servers, high-speed networking, storage connectivity, firmware, drivers, and out-of-band management. You will serve as a senior technical escalation point for complex infrastructure incidents affecting GPU clusters, compute nodes, networking, storage, and supporting management services. You will work closely with Network Engineering, Data Center Operations, Deployment Engineering, vendors, and customer technical teams to restore service, reduce recurring failures, and improve the reliability of production HPC environments. This position requires strong Linux, and hardware troubleshooting skills, hands-on experience with enterprise server hardware, and the ability to diagnose issues across the full infrastructure stack-from physical components and firmware through the operating system and cluster-management layer.

Requirements

5+ years of experience in Linux systems administration, HPC infrastructure, AI infrastructure, data center operations, or a related field.
Advanced Linux administration and troubleshooting experience in production environments.
Hands-on experience supporting enterprise server hardware and bare-metal compute systems.
Experience supporting NVIDIA GPU infrastructure, accelerated computing platforms, or large-scale GPU clusters.
Strong understanding of server hardware, firmware, BIOS, BMC, PCIe, NUMA, memory, storage, and network interfaces.
Experience troubleshooting hardware failures using operating-system logs, BMC logs, vendor diagnostics, and hardware-management tools.
Experience with NVIDIA drivers, CUDA compatibility, DCGM, Fabric Manager, nvidia-smi, or related GPU-management tools.
Working knowledge of Ethernet and high-performance networking technologies such as InfiniBand, RDMA, RoCE, and NVIDIA/Mellanox adapters.
Experience with out-of-band management technologies such as Redfish, IPMI, iDRAC, or iLO.
Experience with one or more automation or scripting technologies, including Python, Bash, or Ansible.
Familiarity with infrastructure monitoring and observability platforms such as Prometheus, Grafana, Datadog, DCGM Exporter, or UFM.
Ability to troubleshoot across hardware, firmware, operating system, network, storage, and cluster-management layers.
Experience working with incident, problem, change, and escalation-management processes.
Ability to work effectively during production outages and high-priority incidents.
Strong written documentation, communication, and customer-service skills.
Ability to work independently while collaborating effectively across distributed technical teams.
Willingness to participate in a 24/7 on-call rotation.

Nice To Haves

Experience supporting NVIDIA HGX, DGX, H100, H200, B100, B200, or similar GPU platforms.
Experience with NVLink, NVSwitch, NCCL, GPUDirect RDMA, PCIe topology, and multi-GPU troubleshooting.
Experience with InfiniBand fabrics, NVIDIA UFM, MLNX_OFED, subnet managers, and InfiniBand diagnostic tools.
Experience supporting MAAS.
Experience with parallel and distributed storage platforms such as Lustre, BeeGFS, Spectrum Scale, Ceph, or enterprise NFS.
Experience supporting Dell, HPE, Lenovo, Supermicro, or NVIDIA enterprise hardware.
Experience with infrastructure-as-code and configuration-management platforms such as Terraform and Ansible.
Experience with Git-based change control and CI/CD workflows for infrastructure automation.
Familiarity with NetBox or similar infrastructure inventory and IP address management platforms.
Experience managing vendor support cases, hardware diagnostics, RMAs, and field-replacement activities.
Experience working in multi-tenant, customer-facing, or SLA-driven infrastructure environments.
Relevant certifications such as RHCSA, RHCE, NVIDIA, Linux Foundation, CompTIA Linux+, CCNA, or equivalent practical experience.

Responsibilities

Administer, maintain, and troubleshoot Linux-based HPC, AI, and GPU-accelerated compute environments across bare-metal, virtualized, containerized, and cloud platforms.
Diagnose and resolve hardware, operating system, networking, storage, driver, and firmware issues affecting production infrastructure.
Lead complex incident resolution, perform root-cause analysis, and maintain operational documentation, runbooks, and support procedures.
Install, configure, validate, and troubleshoot NVIDIA GPU infrastructure, including drivers, CUDA, firmware, and supporting software components.
Monitor and diagnose GPU health, availability, and performance using tools such as NVIDIA-SMI, DCGM, and Fabric Manager.
Investigate and resolve GPU, NVLink, NVSwitch, PCIe, ECC, thermal, power, and multi-GPU communication issues, coordinating hardware replacements as needed.
Administer enterprise Linux environments, including operating system updates, package management, kernel modules, and device drivers.
Troubleshoot system performance, boot failures, kernel issues, networking, storage, authentication, and security-related problems.
Analyze logs and system metrics to diagnose crashes, kernel panics, resource constraints, and hardware-related operating system failures.
Support enterprise server platforms and troubleshoot CPUs, memory, GPUs, NICs, storage controllers, power systems, and other hardware components.
Manage and validate firmware across servers, networking, storage, and GPU infrastructure to ensure compatibility and stability.
Utilize out-of-band management tools for remote troubleshooting, hardware health monitoring, inventory collection, and post-repair validation.
Troubleshoot Ethernet, InfiniBand, RoCE, RDMA, and GPUDirect networking issues within HPC and AI environments.
Diagnose connectivity and performance problems related to VLANs, MTU mismatches, routing, DNS, packet loss, link degradation, and high-availability configurations.
Collaborate on network deployments, upgrades, fabric stability investigations, and validation of infrastructure changes.
Support and troubleshoot enterprise, distributed, and parallel storage platforms, including NFS, Lustre, BeeGFS, GPFS/Spectrum Scale, and Ceph.
Diagnose storage connectivity, mount, performance, latency, permissions, and capacity-related issues.
Validate storage paths and collaborate with networking and storage teams to resolve service-impacting events and optimize workload performance.
Lead and support incident response activities, coordinating troubleshooting efforts and communications during critical outages.
Conduct root-cause analysis, develop corrective action plans, and track recurring issues through resolution.
Manage infrastructure changes by assessing risk, ensuring operational readiness, maintaining ITSM records, and participating in on-call support rotations.
Mentor and support junior engineers and systems administrators through technical guidance and knowledge sharing.
Lead complex troubleshooting efforts and provide expertise during infrastructure changes and critical incidents.
Promote operational excellence through documentation standards, process consistency, technical reviews, and continuous improvement initiatives.