Principal AI Software Engineer

Microsoft•Redmond, WA

10d•$142,800 - $304,200

About The Position

Join Microsoft’s Compute System Architecture (CSA) team within the System Planning and Architecture (SPARC) organization in Azure Hardware Systems & Infrastructure (AHSI). SPARC manages Azure’s hardware roadmap from architecture concept through production. CSA defines and evaluates systems architecture innovations through hardware/software co-design, drawing on workload insights, emerging technology trends, and industry engagement, and advances ideas toward technical readiness for productization. AHSI delivers hardware systems and infrastructure for Microsoft’s cloud services. The Principal AI Software Engineer will lead software prototyping and workload studies to evaluate memory and systems architecture innovations for Azure, with a focus on reducing total cost of ownership (TCO). The role includes characterizing and optimizing LLM inference workloads, especially KV Cache capacity and movement across GPU HBM, host DRAM, CXL memory, SSD, and emerging memory tiers. The engineer will build proof-of-concepts, evaluation frameworks, and performance models; analyze data movement across compute, storage, and networking; and provide data-driven recommendations that influence hardware architecture and industry alignment over a three-to-six-year horizon.

Requirements

Bachelor’s degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

12+ years of systems software experience, such as OS kernel, memory management, I/O stacks, or virtualization, with a demonstrated track record of guiding architecture and software enablement.
10+ years of experience leading significant hardware/software co-design projects involving CPU and/or systems architecture and influencing technical direction.
Deep expertise in Linux kernel internals, memory management, I/O subsystems, NUMA, DMA, and GPU/CPU/storage/network data paths.
Hands-on experience with NVIDIA GPU software stacks, including CUDA, NCCL, GPUDirect Storage (GDS), and GPUDirect RDMA (GDR).
Understanding of AI inference infrastructure, large GPU clusters, inference serving architectures, and workload performance optimization.
Experience characterizing and optimizing KV Cache-intensive workloads, including long-context, agentic, multi-turn, coding, and reasoning workloads.
Experience with inference frameworks such as vLLM, SGLang, and TensorRT-LLM.
Familiarity with KV Cache technologies such as LMCache, SGLang HiCache, cache offload, cache sharing, and memory tiering approaches.
Experience designing or extending inference runtimes, scheduling systems, memory management components, or KV Cache subsystems.
Understanding of disaggregated prefill/decode architectures, distributed inference serving, and multi-node cache-sharing topologies.
Experience with CXL memory expansion, memory pooling, and memory tiering solutions in large-scale deployments.
Software development skills in C/C++, Python, CUDA, and distributed systems.
Skill partnering with and influencing architects, hardware engineers, and software leads.
Ability to manage through ambiguity, bring clarity and results orientation, and engage and energize collaborators and stakeholders.
Collaboration skills, teamwork, and a sense of presumed responsibility.
Verbal and written communication skills, including the ability to engage technical and non-technical stakeholders at all levels.
Experience leading complex projects with respect and integrity, including multiple workstreams spanning different business and technical disciplines.
Intellectual curiosity and passion for learning and deploying new technologies.
Problem-solving skills, analytical capabilities, and attention to detail.

Responsibilities

Lead full-system software prototyping to develop proof-of-concepts evaluating hardware/software co-designed capabilities for reducing memory TCO, including memory tiering, pooling, and overcommit solutions for Azure workloads and deployments.
Develop workload characterization and correlation insights to identify systems optimization opportunities.
Collaborate with workload experts across Microsoft and partner ISVs to engineer TCO-optimized solutions for Azure general-purpose and specialized compute fleets.
Influence hardware architecture and industry alignment for a three-to-six-year timeframe through data-driven analysis, insights, and recommendations.
Lead characterization and optimization of LLM inference workloads, focusing on KV Cache capacity, placement, migration, and utilization across GPU HBM, host DRAM, CXL memory expansion or pooling, SSD, and emerging memory tiers.
Develop proof-of-concepts and evaluation frameworks for AI inference memory-tiering architectures, including CXL pooled memory, memory expansion, context-memory platforms, SSD-backed cache tiers, and hardware/software co-designed approaches.
Design and execute workload characterization studies for agentic, multi-turn, coding, reasoning, and long-context AI workloads, measuring memory consumption, latency, throughput, token efficiency, and system utilization.
Analyze end-to-end data movement across GPU, CPU, storage, and networking subsystems, identifying optimization opportunities involving GDS, GDR, peer-to-peer memory transfers, and distributed inference pipelines.
Develop software prototypes, framework extensions, and instrumentation to evaluate KV Cache offload, prefetching, migration, compression, deduplication, and memory-overcommit techniques.
Build performance models and simulation frameworks to predict the impact of memory hierarchy innovations on large-scale inference deployments.