Senior Hardware Diagnostics Engineer - Infrastructure

TensorWave•Las Vegas, NV

About The Position

TensorWave is building and operating large-scale infrastructure platforms for high-performance AI workloads across multiple data centers, including GPU-intensive systems, high-throughput networking, and rapidly scaling compute clusters. The Senior Hardware Diagnostics Engineer will own the firmware lifecycle and establish how hardware versions are qualified, validated, and safely deployed across the fleet. The role will set standards for validated hardware, build tooling to enforce those standards, and serve as a technical escalation point for complex failures. The engineer will own firmware across BIOS, BMC, NIC, GPU, and storage controllers, including version qualification, approved baselines, staged rollout, and rollback. They will develop validation automation and fleet-scale management integrations, define burn-in criteria, and make fleet-wide changes safe through pre-flight validation, blast-radius control, staged waves, and tested rollback paths. The role also leads hardware and firmware root cause analysis, vendor escalations, and preparation for next-generation AMD platform bring-up. The engineer will mentor the diagnostics engineer on the team and contribute to incident response, runbooks, documentation, and operational best practices. Within the first 90 days, the engineer is expected to qualify and roll out a firmware release using the existing process and identify its shortcomings. By six months, a previously manual validation step is expected to run automatically, and the approved firmware baseline should be identifiable across the fleet without checking nodes individually.

Requirements

At least 6 years of experience in infrastructure engineering, hardware validation, systems engineering, or a closely related field.
Deep hands-on experience with BMCs and out-of-band management such as Redfish, IPMI, iDRAC, or iLO, including scripted interaction.
Direct experience flashing, qualifying, and troubleshooting BIOS, BMC, and device firmware on enterprise servers.
Strong Linux systems skills, including the boot process, kernel and driver behavior, and hardware diagnostics such as dmesg, lspci, ipmitool, SMART, and vendor CLIs.
Ability to interpret low-level hardware signals, including PCIe topology and error output, SEL and sensor data, and thermal and power telemetry.
Python skills for automation against REST APIs, including Redfish, vendor tooling, and internal systems.
A rigorous diagnostic method: forming hypotheses, designing tests that can disprove them, isolating variables, and documenting evidence.
Experience driving hardware issues with vendors through resolution.
Clear technical writing skills for runbooks, RCAs, and vendor escalations.

Nice To Haves

AMD GPU and ROCm experience, including amd-smi, rocm-smi, or ROCm Validation Suite.
GPU and HPC validation at scale, including stress testing, collective communication testing with RCCL or NCCL, and node health frameworks.
Low-level bus and firmware experience, including I2C, SMBus, BMC firmware behavior, or Redfish schema extensions.
Kernel-level debugging, including crash dump analysis.
Ansible experience for fleet operations and Git-based workflows with CI/CD.
Experience with NetBox or other DCIM and IPAM tooling as a source of truth.
Prior OEM, ODM, or hyperscaler experience in hardware bring-up or validation.

Responsibilities

Own the firmware lifecycle across BIOS, BMC, NIC, GPU, and storage controllers, including version qualification, approved baselines, staged rollout, and rollback.
Design and build a validation harness to qualify firmware releases before production, including regression and performance comparisons between versions.
Automate firmware and validation workflows at fleet scale using Redfish, IPMI, and vendor management APIs.
Define burn-in test matrices, thresholds, and evidence-retention practices.
Make fleet-wide changes safe through pre-flight validation, blast-radius control, staged rollout waves, and tested rollback procedures.
Lead root cause analysis of difficult hardware and firmware failures, including PCIe error analysis, sensor and event-log forensics, and thermal and power telemetry analysis.
Own technical escalations with OEM, ODM, and silicon vendors for firmware defects and track fixes through resolution.
Prepare validation and bring-up capabilities for next-generation AMD platforms before hardware arrives.
Mentor the diagnostics engineer on the team and turn recurring findings into runbooks and tooling.
Participate in incident response and root cause analysis.
Contribute to runbooks, documentation, and operational best practices.
Qualify and roll out a firmware release within the first 90 days and identify issues with the existing process.
Automate a previously manual validation step and establish fleet-wide visibility into the approved firmware baseline within six months.

Benefits

Stock options
100% paid medical, dental, and vision insurance for employees
Company Health Savings Account contributions
100% paid short-term and long-term disability insurance for employees
Life and voluntary supplemental insurance options
Other insurance options, including pet and legal insurance
Supplementary health benefits, including discounted virtual healthcare appointments and serious illness support
Flexible Spending Account
401(k)
Employee Assistance Program
Flexible PTO
Paid holidays
Parental leave
Other in-office perks