Senior Platform Reliability Engineer, Fabric and Interconnect

Firmus Technologies • San Francisco, CA

14h • Hybrid

About The Position

The Senior Platform Reliability Engineer, Fabric and Interconnect, owns the reliability of the interconnect and network fabrics that support Firmus’s large-scale AI infrastructure. These include GPU-to-GPU interconnect domains, high-performance fabrics carrying training and inference traffic, and DPU-based host networking that connects compute to the wider platform.

This is a hands-on senior engineering role focused on reliable operations, automation, and continuous improvement. The engineer builds and maintains guarded automation and remediation tooling to make fabric fault response software-driven and self-healing, diagnoses and tunes performance across the interconnect stack, operates DPU-based networking, and executes carefully staged firmware upgrades and capacity changes. The role also provides deep technical leadership during fabric incidents, works with vendors at engineering level, and drives permanent fixes in collaboration with AI Infrastructure.

Requirements

• At least 8 years of experience in high-performance networking and systems engineering, including substantial ownership of production network or interconnect infrastructure in a 24/7 environment.
• Deep operational experience with high-performance GPU interconnect fabrics, such as NVLink and NVSwitch, including domain topology and failure diagnosis.
• Extensive experience with high-performance networking fabrics, such as InfiniBand or RoCE-based Ethernet including NVIDIA Spectrum-X, and knowledge of routing internals and congestion control tuning.
• Experience with DPU- or SmartNIC-based host networking, including offload paths and driver and firmware coordination.
• Experience planning and executing firmware upgrade waves and capacity expansions on production fabrics with defined rollback.
• Strong infrastructure automation and infrastructure-as-code skills, with changes delivered through peer review and progressive rollout.
• Practical scripting or programming experience for operational automation and tooling, such as Python, Go, or Bash.
• Ability to serve as a senior production escalation point, including major incident response, on-call participation, engineering-level vendor escalation, and production of executable runbooks.
• Clear technical judgment and communication skills, including the ability to explain complex fabric failures to engineers and non-specialists.
• Must be based in the San Francisco Bay Area or willing to relocate.

Nice To Haves

• Experience operating fabrics for large-scale distributed training or inference workloads.
• Experience with NVIDIA rack-scale or multi-node GPU systems and their interconnect topology.
• Experience in a multi-tenant service provider, cloud, or colocation environment.
• Knowledge of data center and hardware fundamentals, including cabling and optics, firmware management, and hardware fault workflows.
• Relevant vendor certification.
• A bachelor's degree in computer science, engineering, or a related discipline, or an equivalent combination of relevant experience and training.

Responsibilities

• Own the reliable operation, automation, and continuous improvement of GPU interconnect and network fabrics, such as NVLink and NVSwitch domains, InfiniBand, and Spectrum-X.
• Build and maintain guarded automation and remediation tooling for fabric faults, including fault isolation and fabric reconvergence.
• Diagnose and tune performance across the interconnect stack, from application collective communication to link level, including NVLink, InfiniBand, RoCE, and congestion control.
• Operate DPU-based host networking across the fleet, including offload path configuration and driver and firmware compatibility.
• Execute firmware upgrade waves, fabric expansions, and capacity changes, including production-window ownership, staged or canary rollout, verification against success criteria, and rollback.
• Provide deep technical expertise in fabric and interconnect faults, correlate collective communication failures to physical links, and drive permanent fixes to closure through AI Infrastructure.
• Lead engineering-level vendor escalations by reproducing faults to vendor standards and challenging explanations that do not account for observed behavior.
• Lead technical recovery during major fabric incidents, drive changes to eliminate recurring causes, and participate in the follow-the-sun on-call roster.
• Mentor engineers responsible for frontline diagnosis and document operational procedures, runbooks, and performance results.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed