Software Engineer, Model API Infra

Microsoft AI•Redmond, WA

1d•$119,800 - $304,200

About The Position

As a Member of Technical Staff, Model API Infra, you will own the production API and agentic platform for MAI models. Combining strong software engineering with a deep understanding of how models are trained, evaluated, and served, you will build harness infrastructure ensuring training and production parity, automate the model deployment lifecycle, and create cross-stack observability and tooling to run these services reliably at scale. You will collaborate closely with post-training, evaluation, Foundry, Copilot, and product teams, to continuously deliver new model capabilities to production surfaces.

Requirements

Bachelor's Degree in Computer Science or related technical field AND 5+ years of technical engineering experience with coding in languages including Python, C++, or similar OR equivalent experience.
Experience designing, building, and operating large-scale production distributed systems, cloud services, or developer-facing APIs.
Strong systems design and debugging skills, including the ability to investigate failures across service boundaries.
Experience with deployment automation, orchestration such as Kubernetes, and cloud platforms.

Nice To Haves

Master's Degree AND 5+ years of relevant experience OR equivalent experience.
Experience building model APIs, AI gateways, agent runtimes, agent frameworks, or tool-calling and long-running execution systems.
Experience deploying and operating LLMs or multimodal models through platforms such as Microsoft Foundry, Azure Machine Learning, or custom serving infrastructure.
Familiarity with LLM serving concepts, including streaming inference, token-based capacity, batching, KV cache behavior, model routing, and heterogeneous GPU pools.
Experience evaluating agents on reasoning, coding, tool use, or long-horizon tasks.
Familiarity with post-training, evaluation workflows, and open-source ML frameworks

Responsibilities

Design, implement, and operate scalable API services for text, multimodal, and emerging MAI models, with strong compatibility, versioning, reliability, and performance guarantees.
Build API and systems that maintain consistent model, tool, and harness behavior across training, evaluation, and production serving.
Partner with Foundry, Copilot platforms, inference, research, eval, and product teams to provide consistent interfaces and enable rapid model iteration and delivery.
Build business, API, and model system telemetry that enables cross-system debugging, automated investigations, and data-driven insights.
Build a closed-loop system that automates model deployment and validation.
Integrate functional, safety, quality, and performance evaluations into deployment decisions; debug eval regressions across the stack, from checkpoint and serving config to API and product.
Design and build autoscaling for model pools using latency SLOs, utilization, model characteristics, and capacity constraints.