Principal Software Engineer - AI Infrastructure

Microsoft • Redmond, WA

1d

About The Position

Microsoft’s Cloud + AI (C+AI) organization builds the end-to-end AI stack powering innovation across Azure and Microsoft. Within the Foundry, Inference and Training (FIT) Training Team, this role focuses on infrastructure and scientific methods for training Large Language Models (LLMs), Small Language Models (SLMs), multimodal, code-specific, agentic, proprietary, and open-weight models using open-source training frameworks. The work supports reliable, enterprise-grade agentic workflows and the researchers, product teams, and customers who depend on the platform.

As a Principal Software Engineer, you will own the architecture and technical direction of major training-platform components, taking ambiguous problems through production operation. You will work directly with researchers and product teams to understand needs, reduce friction, and measure outcomes. You will provide technical leadership through design, code, and test-strategy reviews; mentor engineers; and coordinate infrastructure roadmaps across C+AI and partner organizations. The role offers the opportunity to deepen expertise in large-scale distributed systems for AI training and to influence architecture and engineering practices on a fast-moving team.

The team seeks a curious, independent, adaptable, customer-focused problem-solver who learns continuously, writes efficient and well-tested code, debugs complex distributed systems, shares knowledge, and collaborates effectively as priorities evolve. Microsoft describes its mission as empowering every person and organization to achieve more, and its culture as grounded in growth mindset, innovation, collaboration, respect, integrity, accountability, and inclusion.

Requirements

• Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
• 5+ years designing, building, and operating large-scale distributed systems or cloud platforms, with end-to-end ownership of production services. Experience should include Kubernetes-based containerized platforms on Azure, AWS, or Google Cloud; control-plane or backend services managing compute, storage, and networking; and secure multi-tenant designs covering authentication, authorization, workload identity, quota management, and isolation.
• 3+ years building machine-learning training infrastructure, such as GPU cluster scheduling, data pipelines, and checkpointing and storage systems, using frameworks such as PyTorch FSDP, DeepSpeed, Megatron-LM, Hugging Face Transformers and Accelerate, or Ray Train.
• 2+ years training, fine-tuning, or evaluating open-weight models at multi-node scale.
• 2+ years building or operating RL post-training pipelines for LLMs and agentic models using verl, slime, MILES, or comparable frameworks, with understanding of the end-to-end training stack—including pretraining and post-training frameworks, RLEs, rollout inference engines, evaluation, and checkpointing—and how design choices affect throughput, stability, and cost.
• 3+ years serving as an on-call or DRI engineer for production services with defined availability or latency targets, including incident response, postmortems, structured logging, distributed tracing, dashboards, alerting, automated validation, and incident playbooks.
• 2+ years partnering directly with internal or external customers to define requirements, instrument customer experience (such as adoption, satisfaction, or time-to-first-result metrics), and prioritize engineering work by customer impact.
• Experience leading technical design across 2+ teams, driving ambiguous multi-quarter initiatives to completion, and mentoring senior engineers.

Nice To Haves

• Master's degree in Computer Science or a related technical field and 8+ years of technical engineering experience coding in the listed languages, or equivalent experience.
• Bachelor's degree in Computer Science or a related technical field and 12+ years of technical engineering experience coding in the listed languages, or equivalent experience.
• Contributions to open-source machine-learning training projects.

Responsibilities

• Own the architecture and technical direction of major training-platform components for LLMs, SLMs, multimodal, code-specific, and open-weight models, meeting scalability, reliability, security, and cost-efficiency requirements.
• Partner with researchers and product teams across Microsoft to understand training and evaluation needs, translate them into platform requirements, and measure customer experience and outcomes.
• Drive ambiguous, multi-quarter initiatives from proposal through production; define milestones, identify dependencies, and negotiate ownership and delivery commitments with partner teams across C+AI.
• Build on and extend open-source training frameworks, including PyTorch, DeepSpeed, Megatron-LM, and Hugging Face libraries, adapting them for proprietary and open-weight models and contributing improvements to the community where appropriate.
• Design and build production services that meet Microsoft security, privacy, Responsible AI, and compliance requirements, including auditable evidence of compliance.
• Lead debugging and resolution of complex issues in distributed training jobs and production services.
• Participate in the on-call rotation as a Designated Responsible Individual (DRI), lead incident retrospectives, and improve troubleshooting guides, testing, safe deployment, and monitoring.
• Mentor engineers and raise the technical bar through design reviews, code reviews, and ownership of the team's test strategy.
• Champion AI-assisted engineering tools and practices across the software development lifecycle.

Benefits

• Certain roles may be eligible for benefits and other compensation; additional benefits and pay information is available through Microsoft's careers website.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal