Principal Software Engineer, AI Infrastructure
About The Position
The Principal Software Engineer, AI Infrastructure will guide the design and implementation of systems that route AI requests, prioritize traffic, and match demand to available computing capacity across regions. The role focuses on ensuring that services such as Azure OpenAI and Microsoft Copilot remain fast, reliable, fair, and efficient across Microsoft's global infrastructure. The engineer will own the overall design of distributed systems for managing AI traffic and computing capacity, write and review secure, well-tested code, and help turn long-term plans into dependable roadmaps. The role includes safe, measured rollouts, on-call responsibility for critical services, incident reviews, improved monitoring, and collaboration with product, research, infrastructure, Microsoft, and OpenAI partner teams. The engineer will also mentor colleagues, promote engineering excellence, and help the team adopt AI tools throughout software development while following Responsible AI practices.
Requirements
- Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience building distributed systems, with coding in languages such as C#, Java, or Go, or equivalent experience.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.
Nice To Haves
- Master's degree in Computer Science or a related technical field and 8+ years of technical engineering experience building distributed systems with coding in languages such as C#, Java, or Go; or a bachelor's degree in a related field and 12+ years of such experience; or equivalent experience.
- 5+ years of experience running distributed services on public cloud platforms such as Microsoft Azure, Amazon Web Services, or Google Cloud Platform.
- 3+ years of experience with systems that route or schedule traffic, or use optimization methods to allocate computing resources.
- Experience with Python, Docker, and Kubernetes.
- Experience with AI or machine learning infrastructure, such as serving models in production or managing GPU fleets.
- Experience leading projects across 3 or more partner teams, from design through launch and operation.
- Experience being on call for highly available services and measurably reducing incidents.
Responsibilities
- Own the overall design of how AI traffic and computing capacity are managed across regions, and guide design reviews that weigh speed, reliability, scale, and cost.
- Write and review clear, secure, well-tested code for routing, prioritization, and capacity planning, and establish reusable coding practices.
- Work with product and program managers to translate long-term vision into a dependable, step-by-step roadmap.
- Use feature flags and gradual rollouts to test changes safely, measure impact, and roll back quickly when needed.
- Participate in the on-call rotation as a designated responsible individual for critical AI services.
- Lead incident reviews and improve monitoring to detect issues earlier and reduce their recurrence.
- Partner with product, research, and infrastructure teams to understand customer needs as AI models and capabilities launch.
- Coordinate shared dependencies and ensure testing and support paths are ready before launches.
- Help keep services secure, private, and compliant through threat modeling, safe deployment checks, and clear records.
- Mentor engineers and share knowledge about building reliable systems at scale.
- Help the team use AI tools and agents throughout the software development lifecycle, including coding, testing, and live-issue investigation.
- Follow Responsible AI practices and track the quality and productivity impact of AI tools.
- Embody Microsoft's culture and values.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
