Senior Supercomputing Operations Engineer

Microsoft

21h

About The Position

Microsoft Azure’s Artificial Intelligence and High‑Performance Computing (AI/HPC) organization powers some of the world’s largest cloud‑native supercomputers used for frontier AI training, scientific computing, and large‑scale distributed simulations. Our team builds and operates hyperscale GPU clusters that consistently place Azure among global leaders in the Top500, MLPerf, and Graph500 benchmarks. By joining us, you step into the engineering core responsible for ensuring these systems remain reliable, performant, and ready for the next wave of AI innovation.
At this supercomputing scale, reliability and operational excellence are engineering challenges of their own. As a Senior Supercomputing Operations Engineer, you will own day‑to‑day operations of InfiniBand and GPU interconnect fabrics and treating them as a single, mission‑critical reliability domain that directly impacts GPU availability, training throughput, and customer SLAs. You will lead incident triage and mitigation, debug complex fabric‑layer failures, and correlate telemetry across nodes, switches, SM behavior, and GPU subsystems to identify true root causes. Your work will focus on resolving real production incidents at scale, improving operational readiness, and preventing recurrence through better tooling, automation, and deep systems understanding.
You will build and use state‑of‑the-art tools to detect issues proactively, close operational gaps, and improve observability across our fabrics. You will contribute to TSGs, operational playbooks, and escalation guides while partnering with internal engineering teams and industry leading manufacturers to drive meaningful fixes. The solutions you develop and the operational improvements you drive will uplift the reliability of Azure’s largest supercomputing deployments and directly support the most compute‑intensive AI workloads running in the cloud.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Requirements

• Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, OR Java, JavaScript, or Python OR equivalent experience.
• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role.
• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.

Nice To Haves

• Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
• 4+ years of experience operating high performance computing (HPC), artificial intelligence (AI), or largescale distributed systems in production environments
• Handson experience operating interconnect fabrics for HPC, AI, or largescale distributed systems in production
• Strong Linux systems knowledge with demonstrated experience debugging lowlevel infrastructure issues
• Demonstrated ability to reason across hardware, firmware, drivers, and software stacks to diagnose and resolve production issues
• Familiarity with InfiniBand Subnet Manager behavior, including routing, congestion control, and fabric telemetry

Responsibilities

• Act as DRI for InfiniBand and GPU interconnect fabric operations, ensuring GPU availability and AI training stability
• Lead incident triage, mitigation, recovery, and root cause analysis for fabric-related production issues
• Perform deep multi-layer debugging across InfiniBand, Subnet Manager, GPU interconnect, PCIe, and GPU interactions
• Drive operational excellence and prevention by identifying systemic failure patterns and authoring TSGs, playbooks, and escalation guides
• Build and leverage automation, telemetry, and tooling to improve detection, debuggability, and mean time to mitigation

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level