Software Engineer, Model Serving System

Microsoft AI•Redmond, WA

8d•$119,800 - $304,200

About The Position

We're hiring a Software Engineer to join a fast‑moving, high‑ownership team building the next‑generation large language model (LLM) serving system. This role builds and optimizes the end-to-end path to get Microsoft AI (MAI)'s models running reliably in production through Microsoft Azure cloud platform. We work at the intersection of distributed model serving systems, inference engine optimization and integration, benchmarking, and evaluation to make MAI models shippable, measurable, and evaluable.

Requirements

Bachelor's Degree in Computer Science or related technical field AND 5+ years of technical engineering experience with coding in languages including Python, C++, or similar OR equivalent experience.
Experience with LLM inference/serving systems (vLLM, SGLang, TensorRT-LLM, or custom inference engines).
Strong systems/debugging skills and comfort analyzing performance on GPU clusters.
Experience with distributed serving/inference concepts (TP/PP/DP, KV cache, speculative decoding).

Nice To Haves

Master's Degree AND 5+ years of relevant experience OR equivalent experience.
Experience deploying models to production serving platforms (Azure ML/Foundry, Kubernetes, containerized deployments).
Familiarity with multi-adapter serving, disaggregated prefill/decode, or large-scale KV cache management.
Experience with observability tools (Grafana, Prometheus or equivalent, Kusto) and analyzing production telemetry.
Experience building evaluation/benchmarking harnesses for LLMs.
Knowledge of GPU topology, memory management, and modern accelerator hardware.

Responsibilities

Partner with engine teams to integrate, optimize, and deploy models to production serving environments with high reliability.
Help implement and evaluate production serving features (e.g., content provenance/watermarking) while minimizing latency impact.
Profile and optimize for production SLOs (TTFT, TPOT, throughput, QPS) on modern GPU hardware and distributed serving system topology.
Work across platform, API, and research teams to streamline the path from model release to production availability.
Support serving of adapted models (including multi-LoRA scenarios) with careful attention to memory and caching behavior.
Help build tooling to package and validate fine-tuned models for production deployment.
Improve the reliability and repeatability of deployment workflows.
Design and run benchmarks using real usage patterns to measure serving performance under realistic conditions.
Make production endpoints evaluable, identify bottlenecks in evaluation workflows, and improve their speed and reliability.
Build and improve dashboards and telemetry to track utilization, latency, cache efficiency, and SLA compliance.
Translate measurement insights into concrete optimizations with quantifiable impact.