HPC Hardware Engineer

The San Francisco Compute CompanySan Francisco, CA
9hOnsite

About The Position

GPU clusters are some of the most performant computers on the planet. Even smaller clusters by today’s standards would have ranked in the TOP500 five years ago. Our infrastructure team is responsible for architecting and deploying new clusters around the world and keeping them running smoothly. You’ll participate in on-call rotation, deploy new environments, fix issues when they arise, and lean into automation to enable deployments at scale. We’re a small but ambitious team so you’ll be an early contributor helping to shape culture, mentor junior engineers, and learn from our customers.

Requirements

  • You will have 5+ years of experience with hands-on operating, supporting, and scaling at least one HPC or GPU compute cluster in production (ideally >1,000 GPUs, but not required)
  • You deeply understand server hardware fundamentals, including GPUs, NICs, PCIe, memory, thermals, and power
  • You have experience troubleshooting high-speed fabrics such as InfiniBand and/or RoCEv2 Ethernet
  • You’re comfortable debugging performance and reliability issues across hardware, OS, drivers, and networking layers; full-stack.
  • The idea of automating fleet operations (provisioning, monitoring, remediation) excites you — you embrace infrastructure-as-code
  • You appreciate, value, and generate strong operational documentation and runbooks
  • You have the ability and willingness to mentor junior engineers and contribute to team culture
  • You’re open to coming in to our San Francisco office 3–4 days per week
  • You’re open to domestic travel when required

Nice To Haves

  • Familiarity with data center operations including power, cooling, and colo/vendor engagements
  • Strong Linux systems administration experience, including kernel drivers, RDMA stack tuning, and performance analysis
  • Experience with schedulers and orchestration systems such as Slurm and Kubernetes
  • Exposure to virtualization technologies (KVM, QEMU, libvirt)
  • Experience utilizing telemetry pipelines for predictive hardware failure detection
  • Knowledge of distributed training performance (NCCL, GPUDirect RDMA, multi-rail networking)

Responsibilities

  • Participate in on-call rotation
  • Deploy new environments
  • Fix issues when they arise
  • Lean into automation to enable deployments at scale
  • Help shape culture
  • Mentor junior engineers
  • Learn from our customers

Benefits

  • Generous equity grant
  • Team members are offered a competitive salary along with equity in the company
  • Yes, we sponsor visas and work permits
  • We match 401(k) plans up to 4%
  • We offer competitive medical, dental, vision insurance for employees and dependents and cover 100% of premiums
  • We offer unlimited paid time off as well as 10+ observed holidays
  • We offer biological, adoptive, and foster parents paid time off to spend quality time with family
  • We cover lunch daily for employees
  • You can buy as many books for the office as you want
© 2024 Teal Labs, Inc
Privacy PolicyTerms of Service