Principal Software Engineer - CoreAI

Microsoft • Redmond, WA

8d

About The Position

Join our team within CoreAI, where we are building the AI data-plane that powers all LLM inferencing workloads across Microsoft and Azure customers—from cutting-edge startups to Fortune 500 enterprises. Our converged AI fabric delivers inference capabilities for all LLMs in Microsoft catalog, including OpenAI, Anthropic, Mistral, Cohere, Llama, and more.
As a Principal Software Engineer - CoreAI, you will shape the future of one of the largest and fastest-growing services in Azure, foundational to Microsoft’s AI strategy. Our mission is to serve models at scale—reliably, efficiently, and with ultra-low latency—enabling a rich set of AI-powered product experiences.
This is a rapidly evolving space with immense opportunities to learn, innovate, and drive industry-wide impact!

Requirements

• Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, or Java
• OR equivalent experience.
• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role.
• These requirements include, but are not limited to the following specialized security screenings:
• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Nice To Haves

• 4+ years of design and problem-solving experience, with understanding of system performance, scalability, and engineering best practices.
• Understanding of distributed systems specifically in request serving at scale; including high-performance storage, distributed databases, and networking across global-scale infrastructures.
• Experience shipping with high velocity and iterative approaches to track a north star.
• Demonstrated experience in building high-quality, reliable systems at scale.
• Ability to lead complex technical initiatives that span multiple teams and disciplines.
• Customer-obsessed approach to problem solving, with empathy and a drive to deliver impactful solutions.

Responsibilities

• Build and scale the core serving systems and smart request routing and distribution for all LLMs (OpenAI, Mistral, Grok, DeepSeek and many others).
• Learn, innovate and build cutting edge innovations in the AI space collaborating with the best and brightest leading it.
• Ship new product features and improvements at a high velocity.
• Design, implement and deliver AI services to support product offerings for large-scale LLM serving.
• Collaborate closely with product management and partner teams to align technical direction with business goals.
• Coach and grow team members with design, code reviews, and mentorship serving as a role model leading product quality and velocity.
• Engage with customers to gather feedback and resolve complex issues.
• Innovate technical solutions, and patterns that will improve the availability, reliability, efficiency, observability, and performance of products.
• Embody our culture and values

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees