Principal AI Infrastructure Engineer

Highrise AI

About The Position

Highrise AI is seeking a senior, hands-on AI Cluster Integration Engineer to bring large-scale GPU compute clusters from physical installation to validated, commercially ready production. Working alongside senior infrastructure engineers, this person will own critical aspects of cluster bring-up, bare-metal provisioning, advanced systems configuration, performance optimization, and acceptance testing. After deployment, the engineer will provide Tier 3 technical escalation and sustained performance support for operating clusters, focusing on compute-platform readiness and workload performance rather than primary ownership of network switch and fabric configuration. The role supports large-scale AI compute infrastructure across multiple sites. Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting is required. Flexibility to assist with critical incidents or maintenance outside regular business hours is expected. The role is hybrid or remote depending on location. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. Highrise AI is a GPU-native infrastructure platform that delivers scalable GPU compute across dedicated clusters, managed cloud, and confidential environments. It combines bare-metal performance with full-stack orchestration, API-driven access, and hardware-enforced isolation to support AI training, fine-tuning, and production-scale deployment. Highrise also operates third-party GPU hardware for customers and partners. It is a subsidiary of Hut 8 Corp., an energy infrastructure platform.

Requirements

Bachelor's degree in computer science, engineering, or a related field, or equivalent practical experience.
8+ years of hands-on Linux systems engineering, HPC, cloud, or large-scale infrastructure experience, including senior-level troubleshooting.
Proven experience bringing GPU or HPC clusters into production and qualifying systems against performance and reliability targets.
Strong expertise in Linux, automation (Python/Bash/Ansible), bare-metal provisioning, configuration management, and systems diagnostics.
Hands-on experience with NVIDIA GPU platforms, NCCL, Field Diagnostics, and multi-node performance testing.
Practical experience with Kubernetes and Slurm.
Deep technical curiosity and ability to isolate difficult problems across hardware, firmware, operating systems, and distributed workloads.
Strong ownership of outcomes, with a systematic approach to testing, documentation, root cause analysis, and continuous improvement.
Ability to collaborate with experienced peer engineers while driving independent technical decisions and execution.
Clear communication with engineering leadership, vendors, site personnel, and customer-facing stakeholders.
Willingness to travel to data centers for commissioning, acceptance, and complex troubleshooting.
Flexibility to assist with critical incidents or maintenance activities outside regular business hours.

Nice To Haves

Familiarity with high-speed Ethernet, InfiniBand/RDMA, and parallel or distributed storage.

Responsibilities

Lead technical bring-up of materially complete, racked, powered, and cabled AI clusters through production readiness and customer handoff.
Establish and validate system baselines across BIOS/BMC, firmware, GPU drivers, operating systems, and cluster software stacks.
Deploy and optimize automated bare-metal provisioning, imaging, node configuration, and lifecycle workflows at scale.
Configure and troubleshoot advanced Linux and GPU systems, including kernel, driver, accelerator, and host-level performance settings.
Integrate and validate Kubernetes and Slurm environments in partnership with platform, networking, and operations engineers.
Execute NVIDIA Field Diagnostics, L11/L12 validation, burn-in, stability, and hardware health qualification procedures.
Run and analyze NCCL collective communication tests, GPU benchmarks, and distributed workload performance at node, rack, and cluster scale.
Troubleshoot GPU, NVLink/NVSwitch, PCIe, RDMA, and InfiniBand/Ethernet performance issues, partnering with network engineers on fabric-related remediation.
Identify configuration bottlenecks and optimize throughput, latency, scaling efficiency, and cluster reliability.
Own technical acceptance criteria, test evidence, defect tracking, remediation verification, and readiness recommendations for commercial service.
Serve as a senior escalation point for complex production incidents involving compute nodes, GPU software, provisioning, scheduling, and degraded performance.
Perform deep root cause analysis, coordinate vendor engineering escalations, and lead corrective actions for recurring or systemic issues.
Support fleet-level upgrades, configuration changes, cluster expansions, and post-maintenance validation with appropriate change controls.
Develop diagnostic tooling, automation, runbooks, and operational handoff material for SRE, NOC, and site support teams.
Work closely with infrastructure engineering, site operations, OEMs, and software/platform teams to resolve cross-stack issues and meet delivery milestones.
Provide concise technical status, benchmark findings, risks, and acceptance recommendations to engineering leadership and stakeholders.