AI Infrastructure Operations Engineer - Compute & Storage - Highrise AI

Hut 8

6d•Hybrid

About The Position

Highrise AI is seeking a mid-level AI Infrastructure Operations Engineer to provide Tier 2 operational support for large-scale GPU compute and storage platforms. This hands-on role owns technical tickets, restores service, investigates recurring issues, and supports reliable day-to-day operation of production AI clusters. The engineer works with the NOC, on-site data hall technicians, SRE teams, and senior infrastructure engineers to troubleshoot Linux-based compute nodes, GPU systems, storage services, and supporting software. Complex or systemic issues are escalated to Tier 3 engineering with clear diagnostics and supporting evidence. The role covers incident and ticket management; compute and GPU platform operations; storage monitoring and troubleshooting; and operational readiness and continuous improvement. Responsibilities include diagnosing faults, coordinating service restoration, supporting node provisioning and maintenance, investigating Kubernetes and Slurm failures, troubleshooting storage services, assisting with upgrades and expansions, maintaining operational records, and improving runbooks and automation. This is a full-time engineering role supporting AI compute infrastructure across multiple sites. Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting is required. Flexibility to assist with critical incidents or maintenance outside regular business hours is expected. The role is hybrid or remote depending on location. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

Requirements

Bachelor's degree in computer science, engineering, or a related discipline, or equivalent hands-on experience.
3–6 years of experience supporting Linux servers, HPC, cloud, or data center infrastructure in a production environment.
Practical troubleshooting experience with enterprise compute hardware, BMC interfaces (iDRAC/iLO or equivalent), operating systems, and monitoring tools.
Working familiarity with storage technologies such as NFS, distributed filesystems, object storage, and storage performance fundamentals.
Experience using ticketing systems and change management procedures.
Experience with Bash or Python scripting for routine diagnostics and automation.
Structured, evidence-driven troubleshooting and sound judgment about when to remediate, collaborate, or escalate.
Strong ticket ownership, attention to operational detail, and commitment to reliable customer service.
Ability to work effectively across NOC, SRE, hardware, network, storage, and Tier 3 engineering teams.
Clear written and verbal communication in English.
Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting.
Flexibility to assist with critical incidents or maintenance outside regular business hours.

Nice To Haves

Exposure to NVIDIA GPU systems, InfiniBand/RDMA, Kubernetes, Slurm, or AI/HPC storage platforms.

Responsibilities

Own and manage escalated incidents, service requests, and problem tickets from initial technical triage through resolution, escalation, or documented handoff.
Diagnose compute and storage faults using system logs, monitoring dashboards, management interfaces, and approved diagnostic tools.
Assess severity and customer impact, and coordinate with the NOC and site teams to restore service within operational and SLA targets.
Produce troubleshooting records, ticket updates, shift handovers, and actionable Tier 3 or OEM escalations.
Troubleshoot Linux servers, GPU health alerts, boot failures, firmware and driver issues, hardware faults, and resource availability problems.
Support node provisioning, reimaging, configuration changes, routine maintenance, and return-to-service validation using established automation and runbooks.
Investigate Kubernetes and Slurm node or workload failures, working with platform engineers when application- or scheduler-level escalation is required.
Perform first-level analysis of connectivity or GPU communication symptoms and engage network or Tier 3 engineers for deeper fabric and performance investigation.
Monitor and troubleshoot distributed and high-performance storage systems, including capacity, throughput, availability, and client connectivity.
Investigate filesystem mount failures, storage-node or disk alerts, degraded services, and access or performance complaints.
Execute approved recovery, maintenance, and component replacement workflows with storage vendors and data hall technicians.
Support operational tasks across file, object, and block storage platforms as deployed at Highrise sites.
Assist with upgrades, planned maintenance, cluster expansions, and validation activities following established change controls.
Maintain accurate asset, configuration, and ticket records; improve runbooks, troubleshooting checklists, and knowledge articles.
Identify recurring incidents and automate repeatable operational work with scripts and existing infrastructure management tools.
Collaborate with remote engineering teams, on-site technicians, OEMs, and vendors to resolve issues safely and efficiently.
Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting.
Provide flexibility to assist with critical incidents or maintenance activities outside regular business hours.