Senior Software Engineer

Microsoft • ,

2d • $119,800 - $261,000

About The Position

We are looking for a Senior Performance Engineer with deep expertise in High-Performance Computing (HPC) and Artificial Intelligence (AI) networking performance, particularly across InfiniBand-based GPU clusters. The role is a technical leadership position focused on understanding, analyzing, and optimizing distributed workloads at massive scale, often involving tens of thousands of GPUs interconnected via high-speed networks.

The position requires strong familiarity with Message Passing Interface (MPI), NVIDIA Collective Communications Library (NCCL), collective communication algorithms, and transport technologies such as Remote Direct Memory Access (RDMA) over InfiniBand. The engineer will use network-level debugging, topology-aware optimization, and low-latency, high-throughput communication tuning in Linux environments to address challenges across network systems and distributed applications.

Requirements

• Bachelor's degree in Computer Science or a related technical field and 4+ years of technical engineering experience coding in languages including C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
• Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

• Master's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages including C, C++, C#, Java, JavaScript, or Python; or bachelor's degree in a related field and 8+ years of such experience; or equivalent experience.
• Proven history of debugging network performance issues at scale, such as link-level failures, fabric hot spots, and routing asymmetry.
• Expert-level experience with MPI, NCCL, and collective algorithm behavior in multi-node, multi-GPU systems.
• Deep knowledge of InfiniBand architecture, including verbs, RDMA, SHARP, adaptive routing, congestion control, and QoS.
• Familiarity with job schedulers and network-aware scheduling or topology constraints.
• Familiarity with GPU communication stack internals, including GPU Direct RDMA and NVLink topologies.
• Experience tuning performance in transformer training, large-scale AllReduce, or model/data-parallel AI workloads.

Responsibilities

• Diagnose and resolve communication-related performance bottlenecks across large-scale distributed systems.
• Analyze InfiniBand fabric behavior, including congestion, credit starvation, route imbalance, and link-level errors.
• Tune and debug RDMA transport layers, message ordering, path migration, and Quality of Service (QoS) policies on multi-rail or multi-subnet fabrics.
• Develop custom tooling or scripts to analyze fabric telemetry, traffic flows, and job-induced network behavior.
• Collaborate with the operations team to ensure optimal use of fabric features, including adaptive routing, congestion control, and service-level tuning.
• Work with hardware vendors, such as NVIDIA, to evaluate and validate next-generation interconnects and switch firmware.
• Support benchmarking of HPC and AI workloads, including characterization of latency, bandwidth, and collective efficiency under scale.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior