Senior Software Engineer - AI Infrastructure
About The Position
The Senior Software Engineer will work within Microsoft’s AI Infrastructure team, building the AI data plane that powers LLM inference workloads across Microsoft and Azure customers. The converged AI fabric serves models in Microsoft’s catalog, including OpenAI, Anthropic, Mistral, Cohere, and Llama. The role helps shape a large, fast-growing Azure service that is foundational to Microsoft’s AI strategy, with a focus on serving models reliably, efficiently, and with ultra-low latency. The engineer will design and develop distributed systems, routing and scheduling services, algorithms, and control-plane capabilities for managing AI traffic across global infrastructure. The position also involves observability, production operations, fleet efficiency, reliability improvements, technical leadership, mentoring, and contributing to the long-term platform roadmap.
Requirements
- Bachelor’s degree in Computer Science or a related technical field and 4 or more years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
- Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.
Nice To Haves
- Master’s degree in Computer Science or a related technical field and 6 or more years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python; or a bachelor’s degree in Computer Science or a related technical field and 8 or more years of such experience; or equivalent experience.
- Experience with telemetry, metrics pipelines, or resource scheduling systems.
- Familiarity with cloud platforms such as Azure, AWS, or GCP and container orchestration such as Kubernetes or Service Fabric.
- Experience operating large-scale services with stringent reliability and availability requirements.
- Experience with distributed storage systems, telemetry platforms, and real-time data processing.
- Familiarity with AI/ML infrastructure, GPU fleets, inference serving systems, or large-scale model deployment platforms.
- Experience leading cross-team initiatives and driving projects from design through production deployment and operation.
Responsibilities
- Design and implement scalable, reliable, and performant distributed systems for Azure AI traffic management and scheduling.
- Drive architecture and technical direction for routing, capacity allocation, and workload optimization services.
- Develop algorithms and control-plane systems to distribute AI workloads across global infrastructure while meeting latency, availability, and cost objectives.
- Collaborate with partner teams to onboard models, regions, and customer offerings while maintaining operational excellence.
- Build observability and operational tooling to monitor service health, capacity utilization, traffic patterns, and customer impact in near real time.
- Lead investigations into complex production issues and drive improvements to reliability, resiliency, and platform automation.
- Identify and execute opportunities to improve fleet efficiency, reduce operational overhead, and optimize use of premium AI infrastructure.
- Mentor engineers, review designs and code, and raise engineering standards.
- Promote engineering excellence through testing, automation, documentation, and operational best practices.
- Influence long-term platform strategy and contribute to roadmap planning for next-generation AI infrastructure.
Benefits
- Certain roles may be eligible for benefits and other compensation; additional benefits and pay information is available through Microsoft Careers.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
