Senior Infrastructure Engineer

Crusoe•Sunnyvale, CA

3d•$170,000 - $205,000

About The Position

Crusoe is seeking a Software Engineer to join its Hardware Systems Engineering team. The role builds tools, automation, and data platforms that help hardware engineers debug, validate, and support high-performance compute systems at scale. The engineer will own software supporting the hardware lifecycle, from prototype bring-up through large-scale production, to accelerate issue resolution and improve reliability across Crusoe Cloud’s GPU- and CPU-based infrastructure. The role works closely with hardware engineers to understand debugging and validation workflows and make them automatable, while collaborating with cross-functional teams to integrate tools into the production environment. Key areas include provisioning, hardware health monitoring, test frameworks, and data collection. The work supports deployment and operation of sustainable, AI-first compute systems with strong performance and reliability.

Requirements

At least 5 years of professional software engineering experience, ideally building internal tools, infrastructure software, provisioning, test automation, or systems software.
Strong proficiency in Python and at least one systems language (Go, C/C++, or Rust), plus Shell scripting.
Experience designing and implementing automation and test frameworks, including workflow engines, for hardware or large-scale infrastructure.
Experience with BMC/Redfish/IPMI, firmware update automation, or out-of-band management.
Experience with fleet management, provisioning, or large-scale automation tools such as Kubernetes, Ansible, or Terraform.
Experience building data pipelines, logging, telemetry, or observability tooling to analyze system behavior at scale.
Experience working across hardware, firmware, and software teams in multidisciplinary environments.
Strong analytical and problem-solving skills with a data-driven approach.
Excellent communication and collaboration skills, including gathering requirements from engineers and external partners and translating them into usable tools.
Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience.

Nice To Haves

Experience building tooling for GPU clusters, including GPU-to-GPU communication diagnostics such as NCCL tests and NVLink monitoring.
Familiarity with PCIe, InfiniBand, and NVMe/storage concepts and tooling such as lspci, ibstat, nvme-cli, perftest, and fio.
Working knowledge of Linux systems, including kernel interfaces, drivers, device tooling, and low-level system diagnostics.
Familiarity with NVIDIA tooling such as DCGM, NVML, and fabric management software.
Experience supporting both ARM and x86 server architectures.
Background in factory test, manufacturing test software, or hardware qualification workflows.

Responsibilities

Design, build, and maintain software tools and frameworks for hardware development and sustaining, including bring-up, validation, deployment, and production support.
Develop automated test, diagnostics, and health-monitoring frameworks that run across server fleets.
Build tooling, monitoring, and logging systems to help hardware engineers triage and identify root causes of issues across hardware platforms.
Create automated validation and characterization workflows for GPU, CPU, and high-performance compute platforms, including result collection, analysis, and reporting.
Build and operate data pipelines, dashboards, and analytics that surface hardware failure trends, performance regressions, and reliability metrics.
Support end-to-end integration and solution testing by developing test harnesses and CI/CD workflows.
Evaluate and introduce tools, testing methods, and automation approaches that improve engineering efficiency and support sustainability goals.
Provide data-driven insights from tooling and telemetry to influence the hardware roadmap and reliability strategy.