Principal Software Engineering, AI Infrastructure

Microsoft • ,

2d • $142,800 - $304,200

About The Position

The FIT Infrastructure team builds foundational accelerated compute platforms that power large-scale AI training and inference for Microsoft. Its mission is to deliver secure, reliable, and efficient GPU infrastructure for multi-tenant AI systems at global scale, maximizing utilization, availability, performance, and reliability.

This role works at the intersection of cloud infrastructure, distributed systems software, and frontier AI model serving, in close collaboration with Azure Infrastructure, Microsoft Foundry, Microsoft AI, and Microsoft 365 teams. The Principal Software Engineering, AI Infrastructure role focuses on mission-critical infrastructure that powers large-scale AI systems and provides an opportunity to influence cloud GPU platforms used by internal and external customers. The position also offers technical leadership opportunities, including shaping long-term platform strategy and collaborating with experts across hardware, networking, and AI deployment and serving software.

Requirements

• Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Linux, Go, or Python, or equivalent experience.
• Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

• Master's degree in Computer Science or a related technical field and 8+ years of technical engineering experience coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Linux, Go, or Python; or a bachelor's degree in a related field and 12+ years of such experience; or equivalent experience.
• Proven ability to design and operate large-scale production infrastructure with high reliability and performance requirements.
• Demonstrated problem-solving skills and ability to debug complex, cross-layer systems issues.
• Demonstrated technical leadership, including mentoring engineers and driving cross-team architectural alignment.
• Hands-on experience with resource scheduling, preemption, and priority handling on large clusters.
• Advanced collaboration and communication skills, with the ability to work across organizational boundaries.
• Experience with large-scale online inference services.
• Experience building or operating multi-tenant AI platforms in cloud environments.
• Familiarity with inference and/or distributed training stacks, such as NCCL-style collectives, model/data parallelism, or large language models.
• Familiarity with GPU-accelerated computing, such as CUDA, GPU drivers, device plugins, or runtime integration.
• DevOps experience with microservices architecture in a complex infrastructure and operational environment.

Responsibilities

• Develop systems for AI workload management, scheduling, preemption, and failover.
• Develop systems for AI workload serving.
• Design and build GPU-accelerated infrastructure for training and inference workloads across bare-metal, virtual-machine, and containerized environments.
• Build and operate advanced orchestration and resource-governance scenarios using platforms such as AKS and related Kubernetes ecosystem capabilities to enable fair sharing, isolation, and efficient utilization of accelerated resources.
• Design, develop, test, and maintain services written in C# and Golang and hosted on Kubernetes clusters.
• Optimize performance, reliability, and utilization across large GPU fleets, including scale-up and scale-out configurations.
• Drive end-to-end platform features from design through production, including observability, diagnostics, and operational excellence.
• Influence platform architecture and technical direction across teams through design reviews and technical leadership.
• Embody Microsoft's culture and values.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal