Principal Software Engineer - AI Inference and Serving (CoreAI)

Microsoft•Redmond, WA
•$142,800 - $331,200

About The Position

Microsoft Foundry helps developers and organizations build, deploy, and operate AI solutions at scale. This role is on an engineering team building infrastructure to make model training and inference faster, more reliable, and more efficient for customers. The Principal Software Engineer owns an inference service and leads delivery across gateway and runtime boundaries, improving capacity and cost while meeting customer latency and availability expectations.

Requirements

  • Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

  • 8+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • Proven delivery experience in distributed services and model serving spanning at least two connected layers.
  • Strong software engineering fundamentals, debugging skills, and clear cross-team communication.
  • Experience translating customer requirements into service objectives and measurable performance improvements.
  • Experience with containerized services, Kubernetes, and serving runtimes such as vLLM, SGLang, or TensorRT-LLM.

Responsibilities

  • Evaluate prefix caching, prefill/decode disaggregation, speculative decoding, quantization, and tensor/expert parallelism.
  • Preserve correctness for streaming, cancellation, retries, and token accounting.
  • Lead capacity planning, incident prevention, and model onboarding; mentor engineers.
  • Build secure, tenant-isolated systems with automated testing, observability, reliable recovery, and safe releases.
  • Participate in operational support and improve services using production feedback.
  • Preserve model quality and numerical correctness.
  • Use reproducible benchmarks to improve throughput, latency, and cost.
  • Pursue near-linear throughput scaling where workloads permit across agreed GPU and node configurations.
  • Build provider-aware serving stacks and integrate with supported training and post-training stacks for OpenAI, Anthropic, Microsoft AI (MAI), and open-source models.

Benefits

  • Certain roles may be eligible for benefits and other compensation; additional benefits and pay information is available through the linked Microsoft careers page.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service